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Abstract 


This  research  examined  the  magnitudes  of  the  performance, 
supportability,  life-cycle  cost,  schedule,  and  management 
risk  factors  in  current  Republic  of  China  Air  Force  (ROCAF) 
weapon  systems  acquisition  progreuns  and  analyzed  the  risk 
drivers  with  respect  to  those  risk  factors. 

Mid-to-high  level  managers  in  the  ROCAF  and  Chung-Sun 
Institute  of  Science  and  Technology  (CSIST)  were  surveyed  to 
determine  the  extent  to  which  various  risk  factors  were 
contributing  to  program  risk.  A  total  of  fifty-four  (54) 
managers  responded  to  the  survey.  The  results  revealed  high 
risks  associated  with  performance,  supportability,  life 
cycle  cost,  and  schedule,  and  moderate  risk  associated  with 
management.  Two  major  drivers  of  high  risk  in  multiple  risk 
factors  were  also  identified.  First,  ambiguous  requirements 
or  frequent  requirements  changes  drive  performance,  cost, 
and  schedule  risks.  Second,  tight  program  schedules  caused 
by  the  need  for  rapid  system  deployment  contribute  to  per¬ 
formance  and  supportability  risks.  Finally,  recommendations 
for  mitigating  those  risks  and  for  future  studies  were  also 
made. 
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RISK  IDENTIFICATIOM  IN  MAJOR  ffEAPON 
SYSTEMS  ACQUISITION  IN  THE 
REPUBLIC  OF  CHINA  AIR  FORCE 


I.  Introduction 


Background 

In  1982  the  Reagan  administration  enacted  the  817 
Communique  stipulating  the  sale  of  military  equipment  to  the 
Republic  of  China  (ROC)  on  a  yearly  diminution  basis 
(14:74).  In  addition,  the  communist  People's  Republic  of 
China  (PRC)  has  always  sought  to  obstruct  the  purchase  of 
defensive  weapons  by  the  ROC.  For  these  reasons, 
independent  weapon  systems  development  for  national  defense 
was  begun  in  the  ROC  in  early  1980. 

The  goal  was  to  develop  major  defensive  systems  for  the 
armed  forces  independent  of  other  nations.  The  government 
authorized  the  Chung-Sun  Institute  of  Science  and  Technology 
(CSIST),  a  governmental  aeronautic  and  scientific  research- 
and-development  organization,  to  lead  pursuit  of  the  goal  by 
initiating  development  of  the  first  ROC  jet  fighter — the 
Indigenous  Defense  Fighter  (IDF).  The  IDF  is  expected  to 
replace  the  aging  fleet  of  ROC  Air  Force  (ROCAF)  aircraft. 
Ultimately  the  ROCAF  seeks  to  develop  acquisition  experience 
and  to  satisfy  the  operational  user  of  the  IDF — the  ROCAF. 

In  addition,  the  government  wanted  to  shorten  the 


acquisition  process  so  that  systems  could  be  deployed  as 
soon  as  possible.  This  would  help  alleviate  the  expansion 
of  the  force  gap  in  the  Taiwan  Strait  between  the  ROC  and 
PRC  and  eventually  result  in  a  regional  strategic  military 
balance.  This  balance  would  allow  the  ROC  to  pursue 
continued  economic  growth  as  well  as  constructive  and 
peaceful  political  dialogue  across  the  Strait  (15t39). 

Successful  systems  development  requires  an  educated  and 
experienced  work  force  as  well  as  modern  facilities  and  an 
appropriate  budget.  However,  it's  unlikely  that  resources 
will  be  unlimited.  In  fact,  it's  more  likely  that  a  system 
will  have  at  least  one  constraint  which  impedes  the  system 
from  achieving  a  goal  of  higher  performance  (1:4). 

Problem  Statement 

Constraints .  Based  on  years  of  involvement  with  the 
IDF  program,  the  researcher  has  personally  experienced 
several  constraints  (barriers)  to  weapon  systems  acquisition 
in  the  ROCAF.  These  constraints  have  increased  the 
complexity  of  the  program  and  resulted  in  program  risks  that 
are  higher  than  usual.  There  is,  however,  much  the 
government  can  do  to  reduce  the  impacts.  In  short,  the 
constraints  can  be  categorized  as  follows: 

A.  A  majority  of  the  managers  and  engineers  in  both 
the  ROCAF  and  CSIST  lack  relevant  experience. 
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B.  Few  people  are  familiar  with  the  entire  acquisition 
process  and  the  pitfalls. 

C.  In  many  cases,  an  acquisition  strategy  of 
compressed  acquisition  phases  would  be  necessary  to  meet  the 
requirement  for  faster  systems  deployment.  Concurrency  will 
be  inevitable  and  will,  very  possibly,  introduce  additional 
complexities  to  program  management. 

D.  There  is  a  shortage  of  personnel  to  conduct  program 
management . 

Risk.  With  the  IDF  program  the  ROCAF  has  already 
gained  extensive  experience  in  systems  development.  But, 
nonetheless,  an  advanced  training  program  would  ensure  that 
both  management  and  staff  gain  the  appropriate  knowledge 
necessary  to  mitigate  program  risk.  To  alleviate  program 
risk,  the  risks  themselves,  their  impacts,  and  constructive 
solutions  must  be  identified.  The  risks  include: 

A.  High  system  costs. 

B.  Unsatisfactory  system  performance  and/or 
supportabi 1 ity . 

C.  Frequent  engineering  changes  during  development  and 
testing  phases  which  could  cause  schedule  slippage  and 
budget  overrun. 

When  these  phenomena  occur,  the  goal  of  early  systems 
deployment  or  success  of  the  programs  themselves  may  be  in 
jeopardy.  In  short,  however,  in  accord  with  the  policy  of 
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pursuing  independence  of  national  defense,  the  ROCAF  will 
continue  its  weapon  systems  development  in  spite  of  the  many 
inherent  risks  (34t61). 

Sole-Source  Policy 

A  ROCAF  policy  which  might  be  deemed  as  a  problem,  one 
that  differs  from  the  United  States  Air  Force  (USAF)  and  the 
Department  of  Defense  (DoD)  acquisition  policy,  is  the 
selection  of  a  sole-source  prime  contractor  in  the  ROCAF 
versus  competitively  selecting  a  source  as  in  the  U.S.  in 
accordance  with  the  U.S.  Competition  in  Contracting  Act 
(CICA)  passed  in  1984.  According  to  DoDD  5000.1,  the  CICA 
which  was  enacted  to  help  reduce  the  budget  deficit  during 
the  Reagan  administration  (21:1),  stresses  that  "Defense 
systems,  subsystems,  equipment,  supplies  and  services  shall 
be  acquired  on  a  competitive  basis  to  the  maximum  extent 
practicable  as  a  means  of  achieving  cost,  schedule,  and 
performance  benefits"  (19:1-6). 

For  the  ROCAF,  the  reliance  upon  a  single  contractor 
(CSIST)  for  a  major  program  development  may  not  constitute  a 
high  risk  because  of  the  existence  of  acquisition 
strategies,  different  from  those  used  in  the  U.S.,  for 
achieving  program  objectives.  One  of  the  current  ROCAF 
policies  is  to  deploy  systems  as  soon  as  possible;  whereas 
the  DoDD  5000.1  states  that  "schedule  shall  be  subject  to 
trade-off  as  a  means  of  keeping  risk  at  acceptable  levels" 
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(19:1-5).  Also  there  is  a  lack  of  evidence  to  prove  exactly 
when  ineffectiveness  and  inefficiency  of  any  particular 
program  should  be  attributed  to  the  sole-source  environment. 

In  addition,  when  considering  technological  innovation, 
section  2723  of  the  Competition  in  Contracting  Act  says  an 
"agency  head  may  exclude  a  particular  source  upon  a 
determination  that  it  would  be  in  the  interest  o£  national 
defense  in  establishing  or  maintaining  an  essential 
engineering,  research  or  development  capability  provided  by 
an  educational  or  other  nonprofit  institution,  or  federally 
funded  research  and  development  center"  (21:60).  This 
statement  implies  that  the  cheapest  alternative  is  not 
always  the  best  alternative.  Furthermore,  the  apparent 
quantitative  and  qualitative  advantages  attained  through 
competition  and  source  selection;  according  to  General 
Bernard  P.  Randolph,  Commander,  US  Air  Force  Systems  Command 
(7:18);  are  not  necessarily  hard  to  achieve  in  a  single 
source  environment.  Stronger  warranties,  an  expanded 
industrial  base,  technological  innovation,  better  prices  and 
larger  quantities  can  also  be  realized  if  better  management 
and  cooperation  in  all  phases  of  the  acquisition  process  can 
be  achieved.  In  addition,  competition  does  not  guarantee 
those  advantages  stated  above  if  program  management  is  not 
conducted  intensively  and  responsibly.  On  the  other  hand, 
relying  on  single  sources  for  essential  items  could,  in 
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fact,  reduce  future  competition  by  reducing  the  industrial 
base  which  gives  single-source  producers  limited  incentives 
to  reduce  costs  or  invest  profits  in  improved  manufacturing 
or  product  technologies  (8:50). 

In  the  ROCAF,  the  sole-source  contractor  is  a 
government -owned,  non-profit  institution  (the  CSIST),  and 
there  is  no  evidence  to  suggest  that  limited  incentives  have 
resulted  in  higher  costs  for  this  Icind  of  institution. 
Nevertheless,  through  statistical  analysis,  this  research 
will  also  determine  whether  a  government -owned  single-source 
enterprise  has  any  incentives  for  reducing  cost. 

Regardless,  intensive  management  and  production  controls 
should  be  mandatory  to  overcome  possible  pitfalls  in  the 
ROCAF  acquisition  process. 

Justification  of  Research 

There  are  several  reasons  for  doing  this  research: 
First,  "Program  risks  and  risk  management  plans  shall  be 
explicitly  assessed  at  each  milestone  decision  point  prior 
to  granting  approval  to  proceed  into  the  next  acquisition 
phase"  (19:1-4).  In  addition,  "Critical  parameters  that  are 
design  cost  drivers  or  have  a  significant  impact  on 
readiness,  capability,  or  life  cycle  costs  must  be 
identified  early  and  managed  intensively"  (19:1-4).  From 
these  two  points  addressed  in  DoDD  5000.1,  it  is  obvious 
that  assessing  the  program  risks  and  identifying  the  risk 
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drivers  in  advance  of  the  program  phases  are  beneficial  to 
program  success. 

Second,  defense  systems  acquisition  is  too  expensive  to 
learn  every  managerial  lesson  by  trial-and-error . 

Acquisition  experience  is  an  intangible  asset.  However, 
field  studies  and/or  investigations  will  yield  many  valuable 
lessons  to  assist  the  program  manager.  More  specifically, 
the  ROCAF  (or  the  services),  the  CSIST,  and  the  Ministry  of 
National  Defense  (MND)  will  be  able  to  adequately  address 
the  major  risk  areas  as  well  as  the  causative  factors  ahead 
of,  or  during  the  program. 

Third,  "one  indirect  benefit  that  is  a  significant 
by-product  of  a  detailed  risk  assessment  is  the  improvement 
of  program  def inition. . . .Not  only  is  program  definition 
improved  but  all  members  of  the  program  team  develop 
improved  understandings  of  the  total  program  and  their 
places  within  it"  (17:111-4).  Because  the  R(X:AF  and  CSIST 
have  different  goals,  organizational  conflicts  often  impede 
the  progress  of  a  weapon  system  acquisition  program. 

Although  the  unification  of  risk  management  theories  aren't 
necessarily  prerequisite  to  successful  program  management ,  a 
common  understanding  of  acquisition  management  between  the 
ROCAF  and  CSIST  would  result  in  better  risk-management  for 
the  current  and  follow-on  programs.  However,  the  conflicts 
that  occur  between  organizations  and  individuals  during 
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program  management  can  be  reduced  by  conducting  problem¬ 
solving  sessions  to  identify  conflict  causes.  Contemporary 
managers  use  conflict  to  improve  group  cohesion  and  increase 
project  performance  (18:213). 

Fourth,  the  process  of  defense  acquisition  covers  a 
long  period  of  time.  It  would  be  cost  and  schedule 
prohibitive  for  both  tax-payers  and  the  user  to  have  a 
program  terminate  for  default  after  funds  are  obligated  and 
the  acquisition  and  development  has  proceeded.  Because 
penalizing  the  program  manager  won't  recover  the  loss, 
program  management  demands  adequately  trained  personnel  to 
insure  the  selected  source  can  do  the  work.  This  is 
especially  true  for  a  country  with  limited  resources  and  a 
defensive  policy  as  a  national  strategy  like  the  Republic  of 
China.  For  program  management  to  be  successful,  one  must 
understand  the  characteristics  of  the  programs  as  well  as 
the  risks  and  their  drivers  in  advance. 

Fifth,  the  findings  and  recoiranendations  resulting  from 
this  research  could  be  applied  to  the  systems  development 
conducted  by  both  the  Republic  of  China  Army  and  Navy  as 
well  as  the  ROCAF. 

Research  Oblectlve 

Risk  Management  Concepts  and  Guidance  addresses  five 
facets  of  risk  that  are  necessary  to  segment  and  manage  the 
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cost,  schedule,  and  performance  issues  of  a  project: 

.  Technical- (performance  related) 

.  Supportability- (performance  related) 

.  Programmatic- (management  related) 

.  Cost. 

.  Schedule  (6:3-3). 

Cost  and  schedule  risks  are  treated  somewhat  differently 
than  the  other  three  risk  categories  in  that  they  are 
primarily  indicators  of  project  status.  However,  if  they 
are  not  properly  managed,  cost  overrun  and  schedule  slippage 
can  become  major  sources  of  program  risk,  and  ultimately 
lead  to  program  failure  (6:3-3).  The  five  facets  together 
help  define  the  research  scope  and  the  twofold  purpose  of 
this  research: 

A.  To  explore  the  magnitude  of  potential  major  risks 
in  terms  of  performance,  supportability,  life-cycle  cost, 
schedule,  and  management  in  the  ROCAF's  major  systems 
acquisition  process. 

B.  To  know  and  understand  the  risk-drivers  associated 
with  each  risk  factor  in  weapon  systems  acquisition  in  the 
ROCAF. 

Inveetlaatlve  Questions 

The  following  questions  will  be  examined  and  answered 
in  order  to  achieve  the  objectives  of  this  study. 
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Question  1.  What  is  the  magnitude  of  performance  risk 
in  systems  acquisition?  What  causes  the  risk? 

Question  2.  What  is  the  magnitude  of  supportabi 1 ity 
risk  in  systems  acquisition?  What  causes  the  risk? 

Question  3.  What  is  the  magnitude  of  life-cycle  cost 
risk  in  systems  acquisition?  What  causes  the  risk? 

Question  4.  What  is  the  magnitude  of  schedule  risk  in 
systems  acquisition?  What  causes  the  risk? 

Question  5.  What  is  the  magnitude  of  management  risk 
in  systems  acquisition?  What  causes  the  risk? 


Summary 

The  independent  defense  systems  development  program 
supported  by  the  ROC  government  provides  technology  for 
local  Industries,  jobs  for  the  populace,  and  defense  for  the 
country.  Although  the  ROCAF  may  have  additional  reasons  for 
the  independent  development  of  major  weapon  systems,  the 
purpose  of  this  study  is  to  focus  on  identifying  the  risks 
in  the  current  acquisition  process.  This  chapter  has 
briefly  addressed  the  background  of  weapon  systems 
development  in  the  ROCAF,  the  problem  statements,  the 
research  objectives,  and  the  investigative  questions  which 
support  this  research.  Chapter  II  will  address  a  review  of 
the  literature  concerning  risk  and  risk  management.  Chapter 
III  discusses  the  methodology  used  to  execute  this  research. 
Chapter  IV  will  examine  the  data  and  findings  from  this 
research,  and  Chapter  V  will  provide  the  conclusions  and 
recommendations . 
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II.  Llfrature  Rmvlw 


Overview 

Risk  is  the  essence  of  economic  activity.  People 
consider  risks  everyday,  whether  in  their  personal  lives, 
their  businesses.  Risk  management  is  a  disciplined, 
comprehensive  and  continuous  management  process  in  which  the 
applicable  tools  and  techniques  are  used  to  identify, 
evaluate  and  control  the  possibility  of  failure.  Risk 
management  is  dependent  on  individual  perceptions  as  well  as 
the  subjective  and  objective  environments.  These 
environments,  in  turn,  generate  unique  acquisition 
strategies . 

No  matter  how  different  the  acquisitiOi')  strategies 
might  be,  inevitably,  there  is  some  degree  of  risk  in  every 
program  from  "cradle  to  grave"  due  to  uncertainties.  Those 
risks,  depending  on  the  potential  severity  to  the  programs, 
may  result  in  unwanted  consequences,  e.g.,  termination  of  a 
program  for  default.  Termination  not  only  wastes  any 
previous  investment  but  will  result  in  a  great  deal  of 
dissatisfaction  for  the  end  user  as  well  as  frustration  and 
possible  unemployment  for  those  involved  in  the  program. 
Although  the  concept  of  risk  has  been  widely  studied  and 
documented,  there  is  a  general  feeling  that  additional 
knowledge  in  the  areas  of  risk  and  risk  assessment  would  be 
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useful  to  help  assure  satisfactory  program  completion 
(4:75).  Therefore,  the  researcher,  will  review  the  general 
definition  of  risk  in  weapon  systems  acquisition;  and  then 
discuss  the  types  of  risk  that  exist  in  defense  programs. 

In  addition,  the  causes  of  program  risk,  risk  management 
techniques  (including  those  for  risk  identification),  the 
psychology  of  risk,  and  the  concept  of  risk  quantification 
will  be  covered.  Finally,  risk  cannot  be  100%  eliminated; 
therefore,  the  remaining  risk  is  considered  "acceptable 
risk"  and  must  be  adequately  managed.  This  chapter  will 
discuss  acceptable  risk  and  the  part  it  plays  in  systems 
acquisition  and  contractor  performance. 

Review  of  Literature 

Even  though  major  weapon  systems  development  in  the 
ROC  has  matured  over  the  past  twelve  years,  there  is  very 
little  previous  research  associated  with  program  risk 
identification  and  management.  A  search  of  several  business 
periodical  indices,  publications,  and  Air  Force  Institute  of 
Technology  theses  revealed  that  research  of  defense  weapons 
program  risk  as  it  applies  to  the  Republic  of  China  has  not 
been  accomplished. 

Definition  of  Risk 

Any  job  can  be  accomplished  if  there  are  no  constraints 
such  as  time,  budget,  and  manpower.  Unfortunately,  any  real 
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systom  will  likely  have  at  least  one  constraint  (1:4).  In 
fact,  to  conquer  the  challenges  in  the  world  without 
constraints  is  meaningless.  Because  of  constraints,  and 
associated  risks,  the  advancement  of  technology  has  been 
furthered  and  we  have  been  forced  to  develop  solutions  for 
managing  those  risks.  With  risk  management  as  a  normal  part 
of  systems  acquisition,  constraints  that  were  previously 
devastating  appear  to  be  of  little  threat  now.  Does  that 
suggest  constraints  are  gone?  The  answer  is  no-for  two 
reasons. 

The  first  reason  is  the  theory  of  greater  demand,  which 
states  that  customers  will  continually  have  higher 
expectations  and  competitors  will  continually  pressure  the 
status  quo.  Second,  technological  advancement  calls  for 
increased  competence  in  modern  management.  This  is 
definitely  true  in  the  weapon  systems  acquisition  process- 
one  of  the  most  expensive  programs  that  a  country  may  have. 
For  example,  in  the  modern  Electronic  Warfare  (EW)  realm. 
Electronic  Support  Measures  (ESM)  are  used  to  identify  and 
locate  the  enemy's  fire-control  and  guidance  systems.  By 
virtue  of  ESM,  the  enhancement  of  the  probability  of  mission 
success  as  well  as  minimizing  the  loss  of  resources,  e.g., 
equipment  and  lives,  can  be  achieved.  However,  the 
effectiveness  of  the  ESM  could  be  reduced  or  defeated  by  the 
enemy's  Electronic  Counter  Measures  (ECM)  techniques.  ECM 
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may  also  be  minimized  by  advanced  Electronic  Counter -Counter 
Measures  (ECCM). 

Conceivably,  in  order  to  fulfill  the  greater 
requirements  for  national  defense,  a  majority  of  weapon 
acquisitions  are  dealing  with  advanced  techno log ies- 
technologies  with  risks  that  impede  the  program's  ability  to 
achieve  desirable  or  planned  goals.  Therefore,  a  high 
degree  of  managerial  skill  is  required  for  risk  management 
in  defense  acquisition,  and  technical  competence  is 
extremely  important  as  well  (23:141).  In  addition,  the 
constraints,  whether  subjective,  objective  or  both,  present 
many  challenges  that  can  keep  programs  from  being  a  success. 
Knowing  the  nature  of  weapon  systems  development,  it  would 
be  unthinkable  to  manage  programs  by  simply  hoping  risks 
will  take  care  of  themselves.  It  would  also  be  unthinkable 
to  expect  any  resulting  systems  to  be  successfully 
developed,  produced,  and  deployed  with  satisfactory  quality 
and  within  budget. 

According  to  Harold  J.  Schutt,  Associate  Dean,  at  the 
Defense  Systems  Management  College  (DSMC),  "Program 
management  is  risk  management  or,  in  other  words,  the 
program  manager's  job  is  to  manage  risk."  As  a  program 
manager,  before  any  actions  can  be  taken  to  adequately 
reduce  the  risks,  risk  should  be  conceptually  understood  in 
terms  of  the  weapon  systems  acquisition  process.  What  is 
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risk?  Webster's  Dictionary  defines  it  in  generic  terms  ast 

The  possibility  of  loss,  injury,  disadvantage,  or 
destruction:  Contingency,  Danger,  Peril,  Threat,  etc 

(3:1963). 

Obviously,  this  definition  does  not  explicitly  pinpoint 

the  situation  that  a  program  manager  faces.  DoDD  5000.2 

defines  risk  in  much  more  detail: 

A  subjective  assessment  made  regarding  the  likelihood 
or  probability  of  not  achieving  a  specific  objective 
by  the  tisw  established  with  the  resources  provided  or 
requested  (2:15-15). 

Interestingly,  there  are  a  few  key  words  in  the 
definition  which  need  further  explanation  for  a  better 
understanding  of  risk  in  terms  of  acquisition.  These  are, 
subjective  assessment,  specific  objective,  and  resources 
provided. 

A.  Subjective  Assessment.  Because  program  managers 
have  different  propensities  for  taking  risks,  i.e.,  risk 
prone,  risk  neutral,  and  risk  averse;  the  degree  of  risk 
sought  by  two  different  program  managers  might  be 
dramatically  different  even  in  similar  programs. 

Furthermore,  "gut  feel"  risk  assessments  are  often 
performed  by  program  managers.  In  fact,  one  study  showed 
that  many  Aeronautical  Systems  Division  (ASD)  program 
managers  tended  to  perform  qualitative  risk  assessments 
rather  than  quantitative  risk  assessments  (4:77).  (Note: 

ASD  became  the  Aeronautical  Systems  Center  on  1  July  1992). 
Moreover,  there  is  no  evidence  to  indicate  that  any 
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standardized  risk  assessment  methods  were  in  existence  at 
ASD  (4:79). 

The  discrete  judgments  made  by  the  program  managers 
drive  the  program  to  success  or  failure.  Support  for  such  a 
statement  may  be  found  in  the  termination  of  the  A-12 
program  for  the  U.S.  Navy.  The  program  was  terminated  for 
poor  performance  in  1991.  In  this  case,  the  program  manager 
viewed  the  problems  surfaced  by  his  line  managers  as  normal 
and  self  correcting.  He  did  not  perceive  the  need  to  focus 
special  attention  on  the  problems  (20:23-4).  Unfortunately 
the  problems  caused  excessive  cost  overruns  and  unacceptable 
schedule  slippage. 

Quantitative  risk  analysis  might  have  helped  the  A-12 
program.  However,  quantitative  analysis  of  risk  alone 
forgoes  the  opportunity  to  infuse  intuition  to  help  decision 
making  if  there  is  a  high  level  of  uncertainty,  little 
precedent,  limited  time,  or  too  many  plausible  alternatives 
(22:49).  Further,  the  investigation  done  in  1989  showed 
that  under  the  current  ASD  management  climate,  the  majority 
of  mid-to-high  level  ASD  managers  did  not  have  the  time 
necessary  for  performing  more  extensive  risk  assessment 
(4:77).  Clearly,  subjective  assessments  are  integral  to  the 
risk  identification  process. 

B.  Specific  Oblectlve.  A  general  objective  of  weapon 
systems  acquisition  can  be  found  in  the  DoDD  5000.1. 
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An  integrated  manageaent  framework  shall  be  used  for 
translating  broadly  stated  aission  needs  into  stable, 
affordable  acquisition  prograsa  that  saet  the  user's 
needs  and  can  be  sustained  given  project  resource 
constraints  (19:1-1). 

Even  though  specific  objectives  vary  from  program  to 
progreun,  in  most  defense  acquisition  programs,  the 
objectives  essentially  focus  on  fulfilling  the  specified 
requirements  on  time  and  within  the  budget. 

The  seune  is  true  in  the  ROCAF.  The  objective  is  to 
satisfy  the  customers  and  the  Legislative  Yuan  (the  ROC 
Congress).  Quite  frequently,  defense  program  management 
requires  decision-making  and  trade-offs  in  cost,  schedule, 
and  performance.  The  judgmental  skills  necessary  to 
determine  acceptable  level  of  risk  will  be  discussed  later 
in  this  chapter.  In  addition,  there  is  a  trap  that  one 
should  avoid  when  dealing  with  specific  program  objectives. 
The  problem  is  that  many  people  assess  good  program 
management  on  the  basis  of  cost  and  schedule  alone  because 
of  the  difficulty  in  evaluating  the  performance  until  after 
prototype  development.  This  trap  must  be  avoided. 

C.  Resourcas  Provided.  As  stated  previously,  there 
are  normally  limited  resources  provided  for  a  program.  A 
program  manager  will  always  encounter  some  constraints.  The 
most  common  constraints  include:  budget,  schedule, 
technology  and  politics.  Potentially,  if  these  constraints 
are  not  addressed  properly,  they  can  lead  to  overwhelming 
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problems  late  in  the  program.  Knowing  the  relevant 
constraints  early  in  a  program  facilitates  the  establishment 
of  realistic  program  objectives.  Additionally,  knowledge  of 
the  relevant  constraints  facilitates  the  assessment  of 
alternative  choices  when  problems  arise. 

In  summary,  the  definition  of  risk  associated  with 
program  management  includes  the  following.  First,  program 
risk  can  be  assessed  differently  based  upon  the  individual 
risk  propensity.  Second,  a  higher  possibility  of  the 
program  deviating  from  the  original  objectives  equates  to  a 
higher  degree  of  risk.  Third,  a  program  manager  is  given 
resources  such  as,  budget,  time,  and  personnel  to  overcome 
the  difficulties  and  to  satisfy  the  customer.  These  factors 
clearly  indicate  a  program  manager  should  be  intimately 
familiar  with  risk  management  since  he  or  she  is  responsible 
for  the  entire  program. 

Risk  Factors  and  Risk  Drivers  In  Weapon  Systwns  Acquisition 

Risk  Factors  In  Weapon  Svatma  Acquisition.  Risk 
exists  in  whatever  people  desire  to  achieve,  and  objectives 
define  the  existing  areas  of  risk.  For  example,  if  one 
needs  to  reach  Paris  by  next  Monday  to  address  an  important 
seminar,  he  or  she  prepares  the  material  and  has  many 
alternative  routes  for  getting  to  Paris.  However,  he  or  she 
might  be  late  to  the  seminar  (objective)  due  to  many 
possible  delays,  e.g.,  transportation  problems,  loss  of 
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ticket,  loss  of  material,  unaware  of  time  difference.  Jet- 
lag,  sickness,  etc.  This  example  illustrates  that  risks 
will  occur  when  people  pursue  destination.  And  further,  if 
the  process  is  not  adequately  addressed,  the  original 
objectives  may  be  Jeopardixed.  In  weapon  systems 
acquisition,  program  managers  use  resources  to  design, 
develop,  test,  produce,  deploy  and  support  a  system  that  is 
on  time,  with  an  acceptable  quality,  and  within  budget.  In 
systems  acquisition,  the  objectives  are  to  control  schedule, 
cost,  and  performance.  To  assist  in  determining  where  the 
risks  will  most  likely  occur,  the  DoD  4245. 7-M  "Transition 
from  Development  to  Production”  provides  a  template  for 
tracking  the  system  design,  test,  production,  facilities, 
logistics,  and  management.  The  template  is  arranged  in  a 
logical  sequence  from  a  typical  program  manager's  viewpoint 
(5:1-8).  Hence,  program  risks  can  be  categorized  as 
Funding,  Design,  Test,  Production,  Facilities,  Logistics, 
and  Management,  with  funding  influencing  every  other  risk 
area.  Understanding  these  risk  areas  is  fundamental  to  risk 
identification  and  control. 

Risk  Drivers  In  Weapon  Systems  Accrolsitlon.  Program 
managers  and  senior  acquisition  executives  who  are  concerned 
about  the  program  will  always  want  to  know  the  current 
status  of  a  program  and  the  cause  of  any  problems.  They 
will  want  to  know  about  any  factors  which  prevent  management 
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from  achieving  acquisition  objectives.  In  order  to  address 
program  specific  risks,  it  is  important  to  understand  the 
three  distinct  stages  of  science:  classification, 
correlation,  and  ef fect-cause-ef feet  (1:23).  More 
specifically,  science  is  a  process  of  observation,  know-how, 
and  explanation.  The  ultimate  goal  of  applying  these  stages 
is  to  predict  the  outcomes  of  entirely  new  situations. 

Before  the  unwanted  outcomes  become  irreversible,  programs 
should  be  evaluated  by  criteria  that  are  observable  and 
classifiable.  For  weapon  systems  acquisition,  those 
criteria  are  best  described  in  the  “Transition  from 
Development  to  Production"  template.  These  criteria,  if 
studied  in  advance,  will  enable  program  manager  to  recognize 
and  classify  risk  factors  if  and  when  they  occur. 

Based  on  observations,  criteria  that  contribute  to  the 
negative  consequences  in  systems  acquisition  should  be 
identified  for  risk  reduction.  What  can  be  correlated  to 
the  risk  and  its  drivers  in  acquisition?  Before  referring 
to  the  available  template  and  the  other  DoD  publication, 

Beet  Practices,  it  is  important  to  note  there  are  two 
general  sources  involved:  namely,  the  subjective  and 
objective  environments. 

Subjective  environment.  Subjective  environment 
risks  might  be  caused  by: 
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A.  Unqualified  managers  and/or  engineers 
responsible  for  the  program. 

B.  A  lack  of  enthusiasm  for  the  jobs. 

C.  Poor  organizational  structure,  comnunication, 
and  coordination. 

D.  Ambiguously  defined  requirements. 

E.  Inadequate  program  plans. 

F.  Impractical  propensity  for  risk  taking  by  the 
program  manager. 

Oblectlve  environaient .  The  following  objective 
environments  might  also  cause  risks: 

A.  Resource  constraints:  such  as  limited 
manpower,  budget,  and  insufficient  time  for  conducting  the 
necessary  tasks. 

B.  Political  Interruptions. 

C.  Changing  threat. 

Risk  Management 

Sun  Tzu,  the  martial  god  in  Chinese  military  history 
developed  a  doctrine  in  his  offensive  strategy  which  states 
"He  who  has  a  thorough  knowledge  of  his  own  conditions  as 
well  as  the  conditions  of  the  enemy  is  sure  to  win  in  all 
battles"  (10:84).  For  risk  management,  knowing  one's  and 
enemy's  conditions  refers  to  risk  planning  and  risk 
assessment.  Figure  1  shows  the  relationship  of  different 
terms  in  the  risk  management  process. 
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Figure  1.  Risk  Management  Process  (6:5>2) 

The  four  parts  of  the  risk  management  process  can  be 
categorized  into  two  phases,  planning  and  execution.  Risk 
planning  means  to  document  the  risk  management  plan.  In 
most  cases,  program  managers  have  almost  total  freedom  to 
structure  this  document  to  suit  their  situation.  For  proper 
risk  planning,  one  needs  to  master  the  subjective  and 
objective  environments.  Understanding  these  environments 
will  help  to  generate  a  proper  acquisition  strategy  which 
should  prevent  the  program  from  becoming  unmanageable. 

Risk  execution  involves  assessment,  analysis  and 
handling  of  risk.  Risk  assessment  consists  of  both  risk 


identification  and  risk  quantifying.  It  also  appears  that 
risk  identification  has  two  phases,  i.e.,  prior  to  and 
during  implementation  of  the  program.  Knowing  the 
environments,  a  program  manager  should  be  able  to  foresee 
how  his  or  her  program  will  likely  result  if  strategies  are 
not  adequately  considered  and  formulated  at  the  outset  of 
the  program. 

A  surprise  finding  made  in  risk  assessment  in  1989  at 
the  Air  Force  Institute  of  Technology  (AFIT)  was  that  the 
processes/techniques  suggested  by  the  DSMC  were  not  well 
known  by  ASD  mid-level  program  managers  (4:80). 

Nevertheless,  the  study  also  concluded  that  most  of  those 
ASD  mid-level  managers  desired  to  attain  formal  training  to 
help  them  measure  acceptable  levels  of  risk  and  to  decrease 
the  probability  that  they  will  make  inaccurate  decisions 
(4:91-97) . 

According  to  Risk  Management  Concepts  and  Guidance,  the 
execution  portion  of  risk  management  has  seven  steps  as 
outlined  at  the  Table  1  (6:5-2<f): 


23 


Table  1 


Risk  Manaqaawnt  Procadares 

Step _ How/What  to  Do _ 

1.  Evaluate  the 
achievabi 1 ity  of  the 
proposed  project  against 

the  plan. _ 

2.  Identify  the  risk  areas.  1.  Develop  a  structure  to 

systematically  comb  through 
the  progreun  and  issues 
( i . e . ,  Work  Breakdown 
Structure,  checklist) 

2.  Conduct  expert 
interviews. 

3.  Review  analogous  system 
data. 

4.  Evaluate  program  plan. 

5.  Examine  lessons  learned 
document  (Transition 
Template) . 


3.  Quantify  the  risk  areas.  1.  Develop  a  consistent 

scheme  for  rating  risk. 

2.  Assess  the  likelihood  of 
the  risk  occurring. 

3.  Assess  the  intact 
severity  in  terms  of  cost, 

_ schedule,  performance,  etc. 

4.  Document  the  risk  areas.  1.  Develop  and  maintain  a 

management  watch- list. 

2.  Develop  an  effective 
comnunication  scheme  so 
input  from  all  functional 
areas  is  received. 

5.  Conduct  risk  analysis.  Examine  the  results: 

1.  In  terms  of  performance, 
time,  cost. 

2.  By  system/subsystem. 

3.  Of  funding  profiles. 

4.  Based  on  criticality. 

5.  In  terms  of  consistency 
with  analogous  systems. 

6.  Of  "what-if"  analysis. 
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Riak  Manage— nt  Procedures  (Cont) 


6.  Determine  appropriate 
handling  option. 

1.  Avoid  the  risk. 

2.  Share  the  risk  with 
another  party. 

3.  Assiune  the  risk. 

4.  Control  the  risk. 

7.  Implement  the  option. 

Having  a  clear  understanding  of  risk  definition  and  the 
environments  that  potentially  cause  risk  will  not 
necessarily  eliminate  the  risk.  Execution  of  risk 
assessment  and  analysis  are  the  two  major  challenges  for 
achieving  successful  risk  management. 

Appropriate  techniques  suggested  by  the  DSMC 
publication.  Risk  Management  Concepts  and  Guidance  will 
first  be  sumnarized  and  then  risk  identification  techniques 
will  be  reviewed  in  more  detail. 

Sumnary  of  Techniques  for  Defense  Acquisition  Risk 
Assessment  and  Analysis. 

A.  Techniques  for  Risk  Assessment.  The  DSMC 
publication.  Risk  Management  Concepts  and  Guidance 
identifies  several  risk  assessment  techniques: 

1.  Analogous  Comparisons /Lessons  Learned-Examine 
the  "success,  failure,  problems  and  solutions  of  similar 
existing  or  past  programs  (6:5-7)”. 

2.  Expert  Interviews-Obtain  the  judgment  from 
technical  experts  in  the  field  to  a)  identify  those  areas 
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which  may  be  risky  and  b)  the  extent  of  risk  involved  (6:5- 
4) . 

3.  Independent  Technical  Assessment-Obtain  an 
alternative  assessment  of  the  progress  and  risks  of  the 
technical  aspects  of  the  program  from  a  group  of  experts  not 
associated  with  the  developing  organization  (6:5-55). 

4.  Plan  Evaluation-Compare  plans  and  other 
docximentation  to  look  for  contradictions,  inconsistencies 
and  missing  information  (6:5-11). 

5.  Transition  Templates-Conduct  a  comprehensive 
and  disciplined  review  of  a  program  with  the  known  high  risk 
problem  areas  for  the  project  and  the  best  known  solutions 
(6:5-18) . 

B.  Techniques  for  Risk  Analysis.  Additionally, 
the  DSMC  publication  presents  several  risk  analysis 
techniques : 

1.  Decision  Analysis-Develop  a  math  equation 
which  states  the  objective  function  to  be  maximized  or 
minimized  (e.g.  profit,  cost),  the  alternative  choices 
available  to  the  decision  maker,  the  mutually  exclusive  and 
collectively  exhaustive  events  (states  of  nature)  which  may 
affect  the  outcomes,  and  the  probabilities  for  the 
occurrence  of  each  event  (6:5-21). 

2.  Estimating  Relationships-Develop  a 
mathematical  equation  for  the  relationship  between  certain 
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program  characteristics  and  the  cost  or  schedule  to  the 
program  (6:5-26). 

3.  Cost  Risk/Work  Breakdown  Structure  (HBS) 
Simulation  Model -A  computer  simulation  model  can  be  used  to 
aggregate  total  program  costs  based  on  different  probability 
distributions  of  cost  uncertainties  within  lower  level  HBS 
elements  (6:5-37+). 

4.  Cost  Performance  Report  (CPR)  Analysis-Review 
contractor  CPR  to  identify  cost  or  schedule  variances  and 
evaluate  likelihood  of  causes,  potential  impacts  and 
sufficiency  of  remedies  (6:5-54). 

5.  Independent  Cost  Estimates-Obtain  a  total  cost 
estimate  from  individuals,  not  associated  with  the 
developing  organization,  who  use  procedures  and  techniques 
(6:5-56) . 

6.  Life-Cycle  Cost  Analysis-Use  an  existing  or 
modified  computerized  LCC  model,  which  consists  of  a  series 
of  equations  which  compute  program  cost  quickly  for  trade¬ 
off  analysis,  sensitivity  analysis,  production  rate  and 
quantity  analysis,  etc.,  based  on  product  and  program 
information  (6- 5-34). 

7.  Network  Analysis-Develop  a  flow  of  activities 
and  events  which  identifies  relationships,  timing  and 
priorities  (6:5-28). 

8.  Performance  Tracking-Conduct  periodic  reviews 
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of  technical  performance  indicators  compared  to  performance 
goals  and  look  for  breaches  in  pre-established  alert  zones. 
This  technique  advocates  the  use  of  a  Technical  Risk 
Assessment  Report,  which  is  updated  monthly  based  on  working 
level  data  (6:5-45+). 

Risk  Identification.  People  have  different  perceptions 
of  risk.  They  often  not  only  differ  in  quantifying  the 
risks  but  also  in  the  method  used  to  identify  the  risk. 
Several  techniques  for  risk  identification  have  been 
suggested  by  the  DSMC;  they  are:  expert  interviews,  analogy 
comparison,  evaluation  of  the  program  plan,  and  the 
transition  template  (6:5-1+).  Each  technique  will  be 
discussed  in  detail. 

A.  Expert  Interviews. 

1.  Description  of  the  Technique. 

One  of  the  most  critical  elements  or  tasks  in  risk 
assessment  is  that  of  obtaining  accurate  Judgmsent 
frcxn  technical  experts.  Unfortunately,  this  is  an 
area  where  it  is  easy  to  stake  errors  and  therefore 
obtain  information  that  is  inaccurate.  The 
interviewing  of  technical  experts  to  gain 
information  regarding  risk  is  critical  for  two 
reasons.  First,  the  information  identifies  those 
areas  which  are  perceived  as  being  risky  (risk 
identification).  Second,  it  provides  the  basis 
for  taking  the  qualitative  information  and 
transforming  it  into  quantitative  risk  estismtes 
(risk  quantification).  (6:5-4) 

One  might  then  ask  who  qualifies  as  an  expert?  It  is 

frequently  difficult  to  identify  experts.  There  is  no 

measurement  to  accurately  determine  whether  a  person  is  a 
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real  expert.  Expertise  dependent  upon  his  or  her 
experiences  and  achievements.  Though  it  is  important  to 
state  that  one  with  less  experience  is  not  necessarily  less 
capable  than  one  with  more  experience.  Moreover,  an  elder 
manager  who  has  served  a  longer  time  in  program  management 
isn't  always  the  most  competent.  So,  a  safe  way  of 
interviewing  experts  might  be  to  carefully  stratify  the 
population  and  then  take  samples  in  accordance  with 
investigative  interest  or  interview  all  experienced  people 
if  time  permits. 

2.  APDllcabilltv.  The  DSMC  guidance 
suggests  that  Expert  Interviews  are  useful  and  reconmended 
for  all  programs.  Following  this  technique,  it  is  logical 
to  pursue  potential  actions  and  alternatives  as  well  as 
information  pertaining  to  potential  impacts.  With  this 
technique,  it  is  also  possible  to  achieve  higher  cohesion  in 
the  organization  if  brainstorming  or  the  Delphi  Process  can 
be  incorporated. 

^ _ Analogy  Comparison. 

Li _ Description  of  the  Tochnloue.  This 

technique  is  often  called  "lessons  learned”  and  is  self- 
explanatory.  With  this  technique  program  managers  conq>are 
the  characteristic  of  the  new  program  with  past  or  existing 
programs,  and  learn  lessons  from  others'  mistakes.  This  is 
a  feasible  approach,  but,  similarity  between  programs  is 
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essential  to  perforin  the  comparison,  otherwise  misjudgment 
and  erroneous  conclusions  may  result  (6t5-7).  In  other 
words,  the  validity  of  the  data  collected  drives  the 
appropriateness  of  the  technique.  The  analogy  may  rely  on 
similarities  in  technology,  function,  acquisition  strategy, 
manufacturing  process,  etc.  The  key  of  the  technique 
emphasized  by  the  DSMC  guidance  is  to  understand  the 
relationship  between  the  program  characteristics  and  the 
particular  aspect  of  the  program  being  examined.  The 
technique  may  be  time-consuming.  In  addition,  an  older 
system  may  be  somewhat  similar,  but  large  technology  changes 
might  result  in  lessons  that  are  no  longer  valid. 

2.  APDllcabilltv.  This  technique  is  useful 
for  all  phases  and  aspects  of  a  program.  The  DSMC  guidance 
suggests  that  the  technique  is  particularly  valuable  when  a 
new  system  is  primarily  a  new  combination  of  existing 
subsystems,  equipment,  or  components  for  which  recent  and 
complete  historical  data  is  available. 

C.  Evaluation  of  Pr«5|y«i»» 

1.  Description  of  the  Technique.  Plan 
before  doing.  Following  the  plan  cannot  guarantee  success; 
however,  it  can  mitigate  the  risk  if  any  deviations  are 
known  from  the  start.  The  Evaluation  of  Program  Plan 
technique  uses  the  Work  Breakdown  Structure  (WBS)  to  examine 
whether  the  specification  requirements  have  been  addressed. 
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the  schedules  are  feasible,  and  the  cost  is  reasonable. 

These  concerns  should  be  incorporated  into  the  contracts  of 
the  program.  It  is  obvious  that  a  prerequisite  is  required; 
that  is,  the  planning  must  be  as  complete  as  possible  in 
order  to  be  evaluated  properly.  Using  this  technique 
reflects  most  of  the  realities  of  risk  identification 
related  to  the  specific  progreun.  The  relationship  among 
WBS,  SOW,  and  Specifications  is  shovm  in  figure  2. 


Figure  2.  Plan  Evaluation  Technique  (6:5-12) 


2.  APDllcabilltv.  The  DSMC  guidance 
emphasizes  that  this  technique  is  useful  where  technical, 
programmatic,  and  supportability  risk  identification  is 
desired.  The  usefulness  for  cost  and  schedule  risk 
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identification  is  considerably  less;  however,  any  missing 
information  concerning  deliverables  which  would  impact  cost 
and  schedule  could  be  indicated.  The  technique  would  be 
most  applicable  to  the  engineering-and-manufacturing 
development  and  production  phases. 

D.  Transition  Tmnplate. 

1.  Description  of  the  Technique.  This 
template.  Transition  from  Development  to  Production,  was 
built  in  1985  based  upon  actual  experiences  in  DoD's  past 
acquisitions,  and  is  considered  the  most  current  practical 
template  because  its  validity  is  drawn  from  experiences. 

The  major  step  calls  for  individuals  or  groups  to  evaluate 
themselves  in  relationship  to  the  solutions  and  risks 
suggested  in  the  template.  This  template  also  describes 
methods  for  avoiding  or  reducing  the  risk. 

In  addition  to  the  template,  the  US  Navy  published  a 
manual  in  1986  which  is  also  suitable  for  risk 
identification  called  "Best  Practices,  How  to  Avoid 
Surprises  in  the  World's  Most  Complicated  Technical 
Process."  The  manual  provides  detailed  illustrations 
concerning  program  traps  and  their  consequences,  and  also 
provides  a  checklist  for  the  risk  management.  The 
publication  was  derived  from  DoD  Directive  4245. 7-M,  the 
Template. 
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2.  Applicability.  Because  of  the 


acquisition  areas  covered  by  the  teiiq>late  and  the  fact  that 
the  lessons  learned  are  from  actual  experiences,  the  DSMC 
guidance  highly  recommends  the  use  of  this  technique  for 
most  programs,  either  independently  or  in  conjunction  with 
another  technique.  The  template  can  be  used  for  any  size 
program  at  any  phase  of  development  (6:5-19). 

Application  of  the  technique  is  dependent  on  the 
situation  the  program  is  involved  in  as  well  as  the  program 
manager's  experience.  This  technique  is  particularly  useful 
if  the  program  is  a  f irst-of-its-kind,  or  the  risk  analyst 
has  few  experienced  experts  to  consult  and  few  similar 
programs  to  compare.  In  these  cases,  he  or  she  probably  has 
no  choice  but  to  use  either  the  transition-template 
technique  or  evaluate-the-program-plan  technique,  both  of 
which  are  powerful  indeed. 

The  Psychology  of  Risk 

"The  degree  of  risk  existing  in  a  given  situation  is  a 
reflection  of  the  personality  of  the  risk  taker"  (6:4-8). 
People  perceive  risks  differently,  depending  upon  the  nature 
of  the  risk  and  their  individual  experiences.  Risk 
perceptions  are  influenced  strongly  by  issues  of  choice  and 
control,  which  means  that  risks  often  seem  riskier  to  people 
if  they  have  no  control  over  the  source  and  management  of 
the  risks.  Perceptions  of  risks  are  also  influenced  by  the 
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benefits  derived  from  accepting  the  risk  (28:9).  Therefore, 
the  variance  in  magnitude  of  risk  perceived  by  experts  for  a 
given  matter  could  be  very  large  due  to  different 
personalities.  This  subjective  involvement  of  personality 
differences  in  risk  appraisal  is  why  "Expert  Interviews" 
often  conflict  and  produce  inaccurate  information.  The 
psychological  dimension  of  overestimating  or  underestimating 
true  risk  was  confirmed  by  a  study  where  people  estimated 
frequency  of  lethal  events  and  is  summarized  in  Figure  3. 


TSUBntBQUBNCr 


Figure  3.  Judged  Frequency  of  Death  by  Various  Causes 
(30:128) 


In  Figure  3,  note  that  on  the  left,  the  points  are 
systematically  positioned  above  the  "correct"  line  and  on 
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the  right,  systematically  below  it.  Notice  that  homicides 
were  incorrectly  thought  to  be  more  frequent  than  diabetes 
and  stomach  cancer.  Homicides  were  also  judged  to  be  about 
as  frequent  as  stroke,  although  the  latter  actually  claims 
about  11  times  as  many  lives.  Also,  the  incidence  of  death 
from  tornadoes  and  pregnancy  was  greatly  over-estimated 
(30:129).  These  results  Illustrate  the  potential 
shortcomings  of  expert  interviews.  One  study  of  risk 
behavior  in  managerial  decisions  revealed  that,  based  on  the 
REACT  model,  (i.e..  Recognizing,  Evaluating,  Adjusting, 
Choosing,  and  Tracking),  risk  takers  tend  to  underrate 
risks,  while  risk  averters  tend  to  overrate  risks  (31:34). 

Another  interesting  finding  in  the  psychology  of  risk 
arises  from  an  investigation  conducted  in  1960 's.  Stoner 
used  a  personality  assessment  instrument,  the  Choice 
Dilemmas  Questionnaire,  to  test  the  hypothesis  that  groups 
stifle  risk-taking.  Stoner  concluded  that  the 
recommendations  offered  by  individuals  who  are  in  a  group 
situation  were  riskier  than  when  acting  alone  (30:124).  The 
fact  is  that  people's  anxieties  seem  to  be  reduced  in  group 
situations  because  others  are  in  the  same  boat,  or  group 
situations  tend  to  cover  incompetency  of  individuals  when 
involved  in  decision-making. 

Risk  Quantifying 

For  the  purpose  of  controlling  or  diminishing  the  risk. 
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risk  management  should  be  based  on  something  measurable  in 
which  an  "alternatives  comparison"  can  be  conducted.  Hence, 
to  quantify  the  magnitude  of  the  risk  is  highly  recommended. 
Figure  4  illustrates  the  basic  concept  of  quantifying  risk 
magnitude. 


First,  make  a  Judgement,  based  on  experience,  on  the 
likelihood  of  occurrence  of  the  unwanted  outcome  in  terms  of 
probability.  Second,  measure  the  severity  of  impact  based 
on  cost,  schedule,  performance,  or  other  measurable  factors 
or  combinations. 
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Next,  consider  the  dotted  lines  in  the  figure  which 
represent  the  watershed  of  risk  magnitude.  The  positioning 
of  the  boundaries  between  the  different  magnitudes  of  risk 
depend  on  the  program  manager's  propensity  for  risk  taking 
as  well  as  personal  experience.  One  could  draw  the  dotted 
line  that  distinguishes  low  risk  from  medium  risk  in  a  much 
more  convex- to-the-origin  manner  if  he  or  she  is  somewhat 
more  conservative  than  others.  On  the  other  hand,  he  or  she 
might  be  more  of  a  risk-taker.  In  that  case,  the  dotted 
line  would  be  less  convex-to-the-origin. 


Figure  5.  Comparison  of  Risk-Magnitude  Perception 
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Figure  5  illustrates  how  the  magnitude  of  risk 
perceived  differs  individually.  Figures  C  and  D  could  be 
different  perceptions  of  risk  for  different  aspects  of  the 
program  by  the  same  person.  It  could  also  be  referred  to  as 
the  distance  from  consensus  with  regard  to  magnitude  and 
impact  of  risk  on  the  same  aspect  of  the  program  as  viewed 
by  two  people.  For  measuring  magnitude  of  risk  with  respect 
to  the  severity  of  impact  on  different  aspects  of  the 
program,  e.g.,  cost  and  performance,  one  might  view  the 
impact  of  the  occurrence  to  cost  as  more  severe  than  the 
impact  of  the  occurrence  to  performance.  Therefore,  the 
severity  of  impact  is  different  and,  of  course,  the 
magnitude  of  risk  is  different. 

Which  dotted  line  is  drawn  correctly  in  terms  of  risk 
management?  To  a  large  extent,  the  answer  depends  on  the 
particular  program.  DoD  Directive  5000.1  states 
"Acquisition  strategies  and  program  plans  shall  be  tailored 
to  accomplish  established  program  objectives  and  to  control 
risk  ..."  (19:1-4).  All  efforts  including  risk 
management,  are  oriented  toward  the  program  objectives  which 
are  different  for  each  case.  For  example,  if  the  objectives 
of  the  system  development  are  to  deploy  the  system  as  early 
as  possible  and  to  exploit  specific  technologies,  a  safe  way 
to  draw  the  dotted  lines  might  be  shallower  (less  convex-to- 
the-origin)  in  cost  risk  but  steeper  (more  convex-  to-the- 
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origin)  in  schedule  risk.  Because  shallower  dotted  lines 
yield  a  larger  area  between  them  than  steeper  lines,  the 
severity  of  impact  to  the  program  should  be  lessened. 

Acceptable  Risk 

DoDD  5000.2  states  that  all  actions  shall  be  taken  to 
identify,  assess,  and  eliminate  or  reduce  risk  to  an 
acceptable  level  in  selected  areas  (e.g.,  cost,  schedule, 
technical,  producibi 1 ity ,  etc.);  and  the  total  program 
(2:15-15).  When  making  decisions  at  an  individual  level, 
the  choice  of  whether  to  accept  a  risk  is  primarily  a 
personal  decision,  e.g.,  choosing  whether  to  eat  foods  known 
to  have  trace  amounts  of  toxic  substances.  On  a  societal 
level,  the  decision  to  accept  or  not  accept  a  risk  is  more 
difficult  (28:17).  One  actually  does  not  choose  an 
acceptable  risk,  rather  he  or  she  selects  an  alternative 
with  a  certain  degree  of  risk  that  is  expected  to  result  in 
the  lowest  negativity  (12:4).  After  identifying  the  risks, 
the  next  task  would  be  to  decide  whether  the  risk  is 
acceptable  and  why.  No  level  of  risk  will  receive  universal 
acceptance,  but  eliminating  all  risk  is  impossible.  Thus, 
decision-makers  typically  identify  levels  of  risk  that  are 
tolerable  or  acceptable  in  light  of  other  factors  such  as 
the  costs  of  risk  reduction,  or  availability  of  substitutes 
for  that  activity  or  substance  which  poses  the  risk. 
Unfortunately,  this  is  not  an  easy  question  to  answer 
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because  the  magnitude  of  risk  is  perceived  differently  in 
everyone's  eyes,  depending  on  the  propensity  for  risk 
taking.  To  understand  the  nature  of  difference  and  to 
illustrate  the  "best"  way  of  deciding  acceptable  risk,  the 
Cost-Risk  Figure  below  should  be  helpful. 


Safew  Standard 
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Figure  6.  Cost-Risk  Figure  (12:5) 


Figure  6  shows  that  some  risks  are  actually  cost- 
compensable.  The  two  broken- line  indifference  curves  in 
figure  A  represent  the  options  of  taking  higher  risk  in 
return  for  great  reduction  in  cost  at  point  B  or,  to  pay 
higher  cost  for  minimizing  the  risk  at  point  A.  One  might 
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choose  to  avoid  high  risk  or  to  reduce  cost  depending  on  his 
or  her  view  of  the  matter.  However,  choosing  the  proper 
option  is  a  dilemma.  Hence,  these  decisions  require 
sufficient  data  to  assess  the  situation  in  order  to  make  the 
best  choice. 

In  figure  B,  a  safety  standard  is  provided  to  define 
the  region  of  acceptable  risk.  A  standard  may  facilitate  a 
decision,  but  it  isn't  necessarily  a  panacea,  because  every 
option  is  accompanied  by  some  negative  aspects,  such  as 
cost.  Also,  there  are  still  some  people  who  would  like  to 
choose  point  D  instead  of  point  C  to  save  money  since  the 
distance  from  D  to  C  might  be  considered  small  and  in  spite 
of  the  fact  that  D  is  already  over  the  safety  limit. 
Nevertheless,  a  defined  standard  or  baseline  provides  a 
fairly  clear  boundary  to  support  decision-making. 

In  defense  system  acquisition  processes,  the  baseline 
suinmari2es  key  performance,  support,  cost,  and  schedule 
thresholds  for  each  milestone  to  support  a  program  manager's 
management  strategy  and  risk-taking.  In  1986,  The  DoD 
started  baselining  program  objectives  which  were  specified 
in  DoD  Directive  5000.45  (now  incorporated  into  5000.2,  Part 
11).  Even  though  acquisition  stability  may  not  yet  exist 
due  to  changing  requirements  and  other  reasons,  the  baseline 
provides  a  guide  to  program  management  for  trading-off  cost. 
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schedule,  and  performance,  until  the  complexity  of  the 
program  is  determined. 

Another  challenge  which  confronts  program  managers  is 
the  selection  of  suitable  criteria  for  baselining  cost, 
schedule  and  performance.  Baselines  are  adjustable  from 
phase  to  phase  dependent  on  the  changing  acquisition 
strategy.  For  example,  before  the  Engineering  & 
Manufacturing  Development  phase,  program  management  can  use 
a  prototyping  approach  to  assess  cost  and  performance  trade¬ 
offs,  to  define  program  objectives  for  the  development 
baseline,  and  to  determine  contract  specifications  for  phase 
II  (2:5-D-2).  In  later  phases,  production  units  may  be 
available  to  support  baseline  requirements. 

Suninarv 

Risk  is  not  a  welcomed  phenomenon,  but  one  must 
comprehend  the  characteristics  of  risk  before  trying  to 
minimize  its  effect.  Understanding  the  environment  of  the 
defense  program  can  help  the  program  manager  make  an 
educated  choice  for  risk  assessment  and  control  in  the 
weapon  systems  acquisition  process.  However,  implementing 
risk  assessment  and  risk  handling  properly  requires  either 
experience  or  education.  The  techniques  reviewed  in  this 
chapter  provide  a  brief  but  usefui  reference  for  risk 
identification  and  assessment. 

In  this  chapter,  risk  identification,  risk 
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quantification,  and  methods  to  determinlnq  acceptable  levels 
of  risk  have  been  briefly  reviewed.  The  next  chapter  will 
discuss  the  methodology  used  to  conduct  this  research 
effort . 


43 


111.  Methodology 


Overview 

This  chapter  describes  the  research  methodology  used  to 
identify  the  major  risk  factors  in  major  weapon  systems 
acquisition  in  the  ROCAF.  It  presents  the  methodologies 
employed  to  determine  the  degree  of  risk  associated  with 
these  major  risk  factors  as  well  as  primary  causes.  Next 
the  discussion  explains  the  research  population,  research 
sample,  and  data  collection  procedures.  Lastly,  the  chapter 
describes  the  method  used  to  analyze  the  data  collected  in 
this  study. 

Method  Selection 

In  chapter  II,  several  techniques  were  Introduced  for 
risk  identification,  for  example.  Expert  Interviews,  Program 
Comparison,  and  so  forth.  Although  each  technique  can 
effectively  identify  program  risks.  Expert  Interviews  is 
considered  the  most  appropriate  for  ROCAF  acquisition  risk 
identification  for  two  reasons.  First,  expert  interviews 
provides  aggregate  responses  which  represent  a  majority  of 
opinions,  and  that  will  help  to  minimize  the  conflicts 
between  organizations,  and  between  individuals.  Second,  the 
findings  and  recommendations  will  translate  directly  to  the 
ROC  Army  and  Navy  which  have  the  same  acquisition 
environments  as  the  ROCAF.  In  addition,  there  are  several 
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considerations  which  preclude  the  adoption  of  the  other 
techniques : 

A.  Since  major  defense  acquisition  started  just  twelve 
years  ago  in  the  ROC,  there  are  still  very  few  completed 
programs  from  which  to  draw  data  and  lessons  learned.  In 
comparison,  the  United  States  Air  Force  (USAF)  Aeronautical 
Systems  Center  (ASC)  has  completed  many  programs  from  which 
comparisons  can  be  made. 

B.  Although  the  nature  of  systems  engineering 
conducted  in  the  acquisition  process  might  be  the  same,  it 
would  be  simplistic  to  think  that  what  works  in  the  USAF 
will  necessarily  work  in  the  ROCAF.  An  analogy  to  this  can 
be  found  in  the  automobile  industry  where  it  would  be 
simplistic  to  think  the  same  managerial  style  used  in  Japan 
will  necessarily  work  in  the  U.S.  (16:216).  Hence,  one  can 
not  employ  risk  identification  techniques  based  solely  on 
program  comparisons  with  the  ASC.  Additionally  it  is  quite 
possible  that  the  difference  in  cultures  might  result  in 
different  management  styles. 

C.  The  environment  or  the  constraints  might  be 
different.  These  differences  will  introduce  different 
acquisition  strategies  that  are  established  to  achieve 
specific  program  objectives.  Those  strategies,  conceivably, 
will  influence  the  perceived  risk  magnitude  and  the 
selection  of  associated  risk  abatement. 
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D.  Evaluation  of  the  Program  Plan  is  a  technique  used 
for  a  specific  program.  The  intent  of  this  research  is  not 
to  study  a  specific  program's  risk,  but  rather  to  offer  a 
generic  perception  of  risks  that  currently  exist  in  ROCAF 
weapons  acquisition. 

Research  Population 

"Experts"  in  the  ROCAF  acquisition  process  are  defined 
as  mid-to-top  level  managers  who  are  involved  in  day-to-day 
administrative  and  technical  management  and  decision-making. 
These  managers  would  be  equivalent  to  personnel  with  U.S. 

Air  Force  specialty  codes  (AFSC)  27XX  and  0029.  Overall, 
the  population  of  "experts"  can  be  referred  to  as  those  mid- 
to-top  level  managers  distributed  in  defense  programs  within 
the  MND,  services  and  CSIST  who  are  concerned  with  the  day- 
to-day  management  aspects  of  both  major  and  non-major 
programs.  Currently,  the  population  size  consists  of  no 
more  than  two  hundred  managers. 

Sampling  and  Sizing 

A  good  sample  has  both  accuracy  and  precision  (13:278). 
In  order  to  avoid  bias  and  sampling  errors,  the  ideal  survey 
would  include  the  entire  population.  However,  the  8aiiq)le  of 
interest  will  be  drawn  mostly  from  the  ROCAF  because  of  the 
difficulty  of  assessing  both  the  Chinese  Navy  and  Army  and 
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because  of  the  time  limitation.  More  specifically,  the  mid- 
to-top  level  acquisition  managers  in  the  ROCAF  and  CSIST  as 
well  as  MND  will  be  surveyed.  The  number  of  managers  in 
current  ROCAF  acquisition  programs  make  up  approximately 
more  than  one-half  of  the  total  population  due  to  the  large 
scale  of  the  IDF  and  other  ROCAF  programs.  Since  the  number 
of  those  managers  includes  a  large  share  of  the  current 
population,  about  100  out  of  200,  it  is  not  necessary  to 
calculate  a  required  sample  size  specifically.  That  is  to 
say,  basically  the  sample  size  will  include  these  100 
managers . 

Furthermore,  although  day-to-day  experiences  may 
nurture  a  manager's  competence,  there  are  two  reasons  why 
stratified  sampling  will  not  be  used  to  differentiate  or  to 
weight  the  sample.  First,  it  is  difficult  to  determine 
whether  experience  alone  will  directly  determine  a  manager's 
level  of  competence  or  expertise.  Second,  the  survey,  in 
order  to  secure  content  validity  which  will  be  discussed 
shortly,  allowed  the  respondents  to  remain  anonymous. 
Therefore,  the  researcher  cannot  give  weight  to  those 
experienced  managers  who  do  not  identify  themselves. 

Those  managers  with  very  little  experience  (less  than 
six  months),  were  excluded  from  the  sample.  These  new 
managers  were  excluded  because  of  the  time  required  for 
orientation  to  the  acquisition  management  and  the  low 
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utility  of  the  information  which  could  be  obtained  from 
those  individuals.  Incidentally,  lack  of  experience  might 
have  made  those  individuals  more  reluctant  to  complete  the 
survey.  And  inclusion  of  reluctant  respondents  may  produce 
more  error  as  a  result  of  the  poor  quality  of  their 
reporting  (24:60). 

Data  Collection 

The  procedures  and  considerations  concerning  the  data 
collection  are  categorized  into  four  segments  as  follows: 

Scale.  Magnitude  of  risk  measurement  is  subjective. 

An  ordinal  scale  which  allows  the  respondents  to  rank  their 
own  preferences  will  be  used  for  this  study.  A  5-point 
scale  measurement-very  high.  High,  Moderate,  Low,  Very  low- 
will  be  used  rather  than  a  7-point  scale  due  to  the 
difficulty  in  differentiating  the  magnitude  of  extremely 
high  from  very  high  and  extremely  low  from  very  low  in  risk 
assessment.  Besides,  it  is  meaningless  to  distinguish 
extremely  high  risk  from  very  high  if  indeed  both 
assessments  warrant  comparable  levels  of  management 
attention  and  intervention. 

For  risk  assessment,  DoDD  5000.2  states  that  risk 
ratings  of  high,  moderate,  or  low  for  each  major  subsystem 
and  the  overall  system  will  be  used  (2:5-B-2);  which  equate 
to  a  3-point  scale  rating.  Nonetheless,  if  "high  risk"  was 
selected  in  most  subsystems  in  a  program,  the  management 
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would  need  further  distinctions.  These  additional 
distinctions  would  be  helpful  since  available  resources  are 
limited  and  these  distinctions  would  help  to  prioritize  the 
application  of  the  resources  for  risk  management. 

Surveying  Method.  A  questionnaire  was  developed  and 
mailed  to  the  experts.  T/.e  reasons  for  this  strategy  are  as 
follows: 

A.  A  mail-survey  is  faster  and  more  economical. 

B.  A  mail-survey  provides  respondents  with  more  time 
to  consider  the  problems  in-depth  before  answering  the 
questions,  whereas  face-to-face  personal  interviews  have  the 
disadvantage  of  little  time  for  consideration. 

C.  A  mail -survey  allows  respondents  to  mention  the 
real  risk  drivers  because  they  have  the  choice  of  remaining 
anonymous  whereas  personal  interviews  might  make  respondents 
hesitant  to  answer  the  questions  truthfully. 

The  questionnaire  includes  four  segments  of 
investigative  questions.  The  first  segment  asked 
respondents  to  evaluate  themselves  in  terms  of  competency  in 
program  management  and  propensity  for  risk-taking  as  well  as 
other  personal  data  which  will  be  used  to  support  and 
explain  the  findings.  In  the  second  segment,  respondents 
rated  the  specific  risk  factors  in  the  following  areas: 
performance,  supportabi 1 ity,  cost,  schedule  and  management. 
In  the  third  segment,  respondents  ranked  alternative  risk 
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drivers  which  were  based  on  the  "Transition  from  Development 
to  Production"  template  and  the  Navy  Best  Practices 
publication  for  any  risk  factors  which  were  rated  as  very 
high,  high,  or  moderate.  The  last  segment  solicited  any 
additional  comments  on  acquisition  which  could  contribute  to 
this  research.  The  English  version  of  the  questionnaire  is 
attached  in  Appendix  A. 

Validity  Control.  In  order  to  secure  the  content 
validity  of  this  research,  careful  attention  was  placed  on 
the  design  of  the  questionnaires.  This  is  especially 
important  when  applying  the  Expert  Interviews  technique 
(6:5-5).  The  content  validity  of  a  measurement  instrument 
is  the  extent  to  which  it  provides  adequate  coverage  of  the 
topic  under  study  (13:180).  DoDD  5000.2  recommends  using 
the  template  in  DoD  4245. 7-M  "Transition  From  Development  to 
Production"  (2:5-B-3). 

Use  of  the  template,  which  serves  as  a  tool  to 
visualize  the  pitfalls  of  the  DoD  programs,  alleviates  the 
burden  of  risk  management  (6:1-4).  In  addition,  each 
critical  problem  identified  in  the  template  corresponds  to  a 
checklist  and  trap  list  in  the  Best  Practices  which  can  be 
referred  to  when  building  the  questionnaires  (11:2-0).  Most 
importantly,  the  template  and  checklists  published  by  DoD 
were  based  on  real  experience  and  include  the  most  practical 
and  specific  aspects  which  could  result  in  risk.  In 
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addition,  collecting  information  about  self-evaluation  of 
risk  propensities  of  the  respondents  helped  to  modify  the 
measured  risk-magnitudes  which  might  have  been  over-  or 
under-estimated  due  to  the  differences  of  personalities  of 
the  respondents. 

Pilot  Run.  Translation  of  the  questionnaires  from 
English  to  Chinese  was  required.  The  researcher  performed 
the  translation  and  asked  Chinese  students  currently 
studying  at  the  Air  Force  Institute  of  Technology  to  verify 
the  appropriateness  of  the  translation.  This  provided 
additional  validity  to  the  research.  A  translated  Chinese 
questionnaire  is  attached  in  Appendix  B. 

Data  Analysis 

Reviewing  the  objectives  of  this  research  stated  in 
chapter  I,  the  purpose  of  the  study  was  to  try  and  expose 
the  risk  factors  as  well  as  identify  the  risk  drivers  from 
selected  candidates.  The  appropriate  methodology  for  data 
analysis  should  be  discussed  in  terms  of  research 
objectives . 

In  chapter  I,  the  purpose  of  this  research  was 
introduced  as  the  following: 

A.  To  explore  the  magnitude  of  potential  major  risks 
in  terms  of  performance,  life-cycle  cost,  schedule, 
management  and  supportability  in  the  ROCAF's  major  systems 
acquisition  process. 


51 


B.  To  know  and  understand  risk  drivers  that  cause  the 
associated  risk  in  weapon  systems  acquisition  in  the  ROCAF. 

For  the  first  objective,  descriptive  statistics  will  be 
used  to  measure  the  magnitude  of  the  risks.  These  results 
will  reveal  the  magnitude  of  risk  in  performance, 
supportability,  schedule,  life-cycle  cost  and  management  in 
weapons  acquisition  because  of  the  large  sample  size  (24:7). 
More  specifically,  a  pictorial  representation  showing 
relative  frequency  distributions  of  magnitude  can  be 
constructed  using  a  histogr2un.  The  questionnaire  results 
provide  the  frequency  for  a  specific  risk  factor.  A 
qualitative  measurement  can  be  made  with  a  horizontal  line 
of  equal  interval.  Then,  one  needs  only  to  construct 
rectangles  with  heights  proportional  to  the  frequencies 
(25:9+).  Thereby,  a  clear,  easily  understandable  relative 
distribution  of  the  risk  magnitude  of  performance, 
supportability,  and  so  forth  can  be  presented.  However, 
it's  believed  that  before  drawing  conclusions  about  the  risk 
magnitudes  of  the  risk  factors  based  on  descriptive 
statistics,  a  careful  study  of  the  risk  propensities  of  the 
respondents  will  be  necessary.  This  allows  one  to  draw  a 
conclusion  because  the  true  risk  measured  of  a  risk  taker 
will  be  somewhat  different  than  that  measured  by  a  risk 
avoider  even  if  the  magnitudes  of  risk  are  equally  rated. 

For  the  second  research  objective,  identifying  the 
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causes  of  the  risks,  nonparametric  statistics  will  be  used 
to  determine  which  factors  cause  the  most  risk.  There  are 
two  reasons  for  adopting  this  method: 

First,  nonparametric  procedures  forgo  the  traditional 
assumption  that  the  underlying  populations  are  normal 
(21:1).  The  characteristics  of  the  ordinal  scale  used  to 
rank  the  causes  of  specific  risk  and  the  distribution  of 
those  selected  causes  of  risk  can  not  always  be  considered  a 
normal  distribution.  Moreover,  the  selected  causes  of 
specific  risk  might  be  somewhat  related  to  each  other,  for 
example,  if  the  primary  reason  of  resulting  performance  risk 
was  a  poorly-defined  requirements;  the  unclear  requirements 
might  have  resulted  because  of  limited  time  or  because 
unqualified  personnel  were  responsible  for  translating  the 
mission  needs.  On  the  other  hand,  the  secondary  risk  driver 
causing  performance  risk  could  be  an  overall  tight  schedule 
for  conducting  the  task  necessary.  Therefore,  those  primary 
and  secondary  risk  drivers  chosen  are  somewhat  related  to 
each  other,  and  hence,  they  were  not  always  independent. 

Second,  nonparametric  procedures  are  applicable  in 
where  the  actual  magnitudes  of  the  observations  are  not 
required,  but  rather,  their  ranks  (21:1). 

These  reasons  strongly  support  the  selection  of 
nonparametric  tests  for  this  research  objective. 
Specifically,  the  Friedman  Test  is  the  most  desirable 
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approach  to  test  the  equality  among  the  "treatments".  The 
assumptions  and  procedures  for  conducting  the  Friedman  test 
are  as  follows: 

Friedman  Test.  In  a  two-way  array,  the  ranks  can  be 
shown  in  the  diagram  below: 


I  < - TreatoMnts - >  I 
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Risk  drivers  can  be  considered  as  treatments,  and 
respondents  as  block.  Rjj  is  the  rank  of  the  treatments  in 
the  ith  block. 


K.  AssuBiDtionB . 

1.  The  n  k-variate  random  variables  are  anitually 
independent.  (The  results  within  one  block  do  not 
influence  the  results  within  the  other  blocks.) 

2.  Within  each  block  the  observations  may  be 
ranked  according  to  criterion  of  interest 
(26:299) . 

B .  Procedures . 

1 .  Hypotheses . 

H.:  Bach  ranking  of  the  random  variables  within  a 
block  is  equally  likely  (i.e.,  the  treatsmnts  have 
identical  effects). 
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H^:  At  least  one  of  the  treataente  tends  to  yield 
larger  observed  values  than  at  least  one  other 
treatsient  (26:300). 

2.  Test  Statistics: 

mm, 191) 

Where  R  j  is  the  sum  of  ranks  corresponding  to  the  Jth 
treatment.  It  can  be  expressed  as  the  equation  shown 
below: 


Aj-gAij  (2X27:196) 

3.  Decision  rule.  Reject  the  null  hypothesis  at 
approximate  level  a  if  K*  is  greater  than  or  equal  to 

1),  where  upper  a  percentile  for  a  chi- 

square  distribution  with  k-1  degrees  of  freedom  (27:197). 
Actually  the  chi-square  distribution  only  approximates  the 
distribution  of  K*,  but  the  approximation  improves  as  n,  the 
number  of  blocks,  gets  larger  (28:266). 

4.  Conduct  multiple  comparison.  When  the 
Friedman  test  rejects  the  null  hypothesis,  pairwise  multiple 
comparisons  can  be  constructed  to  test  and  determine  if  any 
two  interested  treatments  are  significantly  different  at 
overall  level  a,  if 
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I  I  /6 


(3)(27il97) 


Where  a'“2a/k(k-l)  and  a’ /2«l-4(Zjji /2 >  (27:197).  More 
specifically,  l'-Z^*/2  Z  value  at  a  where  a«l-a'/2. 

Overall  a  is  a  family  confidence  level  that  covers  all 
treatments,  whereas  a*  is  the  significance  level  for  any  two 
treatments  comparison.  For  example,  if  respondents  ranked 
4-treatment  questionnaire,  and  if  an  overall  a  of  0.06  is 
used,  the  treatment  comparison  will  then  have  a  a' >0.01,  or, 
99%  confidence  interval  because  for  4  treatments  there  are  6 
combinations  (4*(4-l)/2=6) ,  hence  o*=(2*0.06)/[4*(4- 
1)1=0.01. 

In  addition  to  rating  risk  magnitude  and  ranking  risk 
drivers,  comments  on  risk  and  why  the  risk  drivers  existed 
were  also  solicited;  from  which,  both  findings  and 
recommendations  were  made. 

Research  Limitation 

As  stated  previously,  risk  assessment  is  dependent  upon 
one's  insight.  It  is  also  difficult  to  judge  whether  the 
manager  is  a  true  expert  even  though  he  or  she  has  many 
years  of  defense  program  management  experience.  Moreover, 
even  if  the  manager  is  an  expert,  experts  are  fallible 
(14:33).  These  are,  in  fact,  inevitable  dilenmas  when 
conducting  Expert  Surveys.  The  drawn  sample  might  contain 
some  non-experts  even  though  those  with  has  less  than  six 
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months  of  management  experience  were  omitted.  This  dilemma 
occurs  frequently  in  the  current  ROCAF  acquisition 
environment  because  middle-to-high  level  manager  are  often 
too  short  of  their  tour  lengths  to  see  the  program  through 
to  completion. 

Summary 

Chapter  III  has  discussed  the  population,  seunple,  and 
methodologies  used  for  risk  identification  and  to  collect 
and  analyze  data;  as  well  as  the  limitation  of  the  research. 
The  next  chapter  will  show  the  results  of  the  investigative 
questions . 
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IV.  Data  AnalYBlB  and  Flndlnaa 


Introduction 

This  chapter  presents  the  results  obtained  by  applying 
the  methodology  described  in  chapter  III  regarding  the  risk 
magnitudes  of  performance,  supportability,  cost,  schedule, 
and  management;  as  well  as  the  risk  drivers  associated  with 
those  risk  factors.  Overall  pictorial  presentations  for 
risk  propensities,  self-evaluation  of  management  competency 
of  the  respondents,  and  the  risk  magnitude  of  specific  risk 
factors  will  be  shown.  These  will  be  followed  by  the  result 
of  the  Friedman  test  which  was  employed  to  investigate 
particular  risk  drivers.  Lastly,  comments  on  systems 
acquisition  management  obtained  from  the  additional  comments 
portion  of  the  questionnaires  will  be  summarized. 

Review  of  Invest iaatlve  Questions 

As  stated  in  Chapter  I,  the  following  questions  will 
be  examined  and  answered  in  order  to  achieve  the  research 
objectives,  namely  the  characterization  of  the  risks 
resident  in  ROCAF  systems  acquisition. 

Question  1.  What  is  the  magnitude  of  performance  risk 
in  systems  acquisition?  What  causes  the  risk? 

Question  2.  What  is  the  magnitude  of  supportability 
risk  in  systems  acquisition?  What  causes  the  risk? 

Question  3.  What  is  the  magnitude  of  life-cycle  cost 
risk  in  systems  acquisition?  What  causes  the  risk? 
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Question  4.  What  is  the  magnitude  of  schedule  risk  in 
systems  acquisition?  What  causes  the  risk? 

Question  5.  What  is  the  magnitude  of  management  risk 
in  systems  acquisition?  What  causes  the  risk? 

Sample  Description 

The  current  research  population  in  the  SOC  defense 
acquisition  process  is  relatively  small  compared  with  the 
U.S.  DoD. ,  and  approximately  two-thirds  of  the  population 
are  responsible  for  ROCAF  major  programs.  The  researcher 
only  sent  questionnaires  to  those  mid-to-top  level  managers 
who  were  involved  in  the  IDF  progreun  due  to  the  difficulty 
in  accessing  the  other  services  and  because  of  time 
limitations.  In  addition,  to  preclude  unqualified 
responses,  the  names  of  extremely  young  managers  with  less 
than  six  months  experience  were  removed  from  the 
questionnaire  mailing  list.  Consequently  72  experts  were 
surveyed,  and  54  of  them  responded  to  the  questionnaires  in 
a  timely  manner. 

Data  Analysis 

Classification  of  the  Respondents.  Before  proceeding 
to  analyze  the  data,  the  reader  should  understand  that  this 
research  did  not  associate  responses  with  the  respondents' 
organization.  Although  it  is  hoped  that  through  this 
research  a  mutual  understanding  of  program  risk  and  risk 
management  for  both  the  ROCAF  and  CSIST  can  be  achieved,  it 
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was  felt  that  differentiating  respondents  by  organization 
would  likely  increase  the  friction  between  the  ROCAF  and 
CSIST.  This  research  attempts  to  understand  the  respondents 
in  terms  of  their  risk  propensities  and  other  self- 
evaluation  information,  which  were  used  to  analyze  the  risk 
factors  and  risk  drivers  perceived  by  the  respondents  in 
their  questionnaires.  For  example,  if  a  majority  of  the 
respondents  were  risk  avoiders,  and  the  risk  magnitude  on  a 
specific  factor  they  measured  was  perceived  as  high  risk, 
then  the  "true  risk"  (population  mean)  might  be  somewhat 
lower  than  the  perception,  since  this  kind  of  respondent 
tends  to  overestimate  risk  magnitude.  The  figures  in  this 
chapter  will  show,  respectively,  the  ratio  of  questionnaire 
mailings  versus  responses,  risk  propensity,  and  management 
competency  of  the  respondents. 


60 


S  I  I  ■  I  ■  m 
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MailiiigB  RespoBBeB 

Figure  7.  Acquisition  Risk  Survey  Participation 

Figure  7  shows  that  36,  31,  and  5  questionnaires  were 
mailed  to  the  ROCAF,  the  CSIST  and  MND,  respectively. 
Although  it  isn't  significantly  different,  the  CSIST  had  a 
slightly  higher  response  rate  than  the  ROCAF  and  MND.  The 
overall  response  rate  was  75%  with  64.8%  for  the  ROCAF, 
93.7%  for  the  CSIST,  and  0%  for  the  MND. 


Risk  Riak  Risk 

Ikker  Nettnl  Avoidet 


Figure  8.  Risk  Propensities  of  The  Respondents 

Figure  8  shows  the  risk  propensities  of  the 
respondents.  Notice  that  the  majority  (59%)  of  respondents 
are  risk-neutral,  37%  are  risk  avoiders,  and  only  4%  of  the 
respondents  are  risk  takers.  The  risk-propensity 
distribution  implies  that  the  risk  magnitudes  they  reported 
require  no  modifications.  It  is  probable  that  the  true 
magnitude  of  the  risk  may  be  overstated  by  the  37%  of 
respondents  who  are  characterized  as  risk  avoiders. 
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Figure  9.  Management  Competency 

Figure  9  shows  the  self-'evaluated  management  competency 
of  the  respondents.  Only  one  respondent  feels  the  day-to- 
day  management  Is  very  difficult,  and  any  types  of  training 
won't  help  to  alleviate  the  frustration.  A  majority  of  the 
respondents,  48  out  of  54,  or  89%  feel  they  are  competent  or 
moderately  competent  in  their  jobs.  Competent  means  that 
although  there  are  some  problems,  overall  day-to-day 
management  and  decision-making  are  basically  sound. 
Moderately  Competent  means  that  although  occasional  problems 
result  in  some  discouragement,  the  day-to-day  management  and 
decision-making  are  not  too  difficult.  This  explains  why 


many  of  the  managers  don't  feel  further  training  is  needed, 
despite  the  fact  that  none  of  them  evaluated  him  or  herself 
as  "Very  Competent"  in  program  management.  The  findings 
will  be  further  discussed  at  the  end  of  this  chapter. 

Investigative  Questions  and  Findings 

Each  investigative  question  is  composed  of  two  related 
questions  in  which  the  first  question  investigates  the 
magnitude  of  the  risk  factors:  performance,  supportabi 1 ity , 
life-cycle  cost,  schedule,  and  management;  and  the  second 
question  reveals  the  risk  drivers  which  result  in  a  moderate 
or  higher  severity  of  the  risk  factor. 

Performance  Risk.  Investigative  Question  1.  What  is 
the  magnitude  of  performance  risk  in  systems  acquisition? 
What  causes  the  risk? 

A.  What  is  the  magnitude  of  performance  risk  in 
systems  acquisition?  To  answer  the  question,  the 
respondents  rated  performance  risk  on  a  five-point  scale  of 
measurement.  The  suiranarized  raw  data  can  be  seen  in 
appendix  C.  The  histogram  below  shows  the  risk-magnitude 
distribution  for  the  respondents. 
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Figure  10.  Performance  Risk 


There  were  5  out  of  the  54  respondents  (9.3%)  who  rated 
the  performance  risk  in  the  current  acquisition  environment 
as  Very  High,  23  respondents  (42.6%)  rated  risk  as  High,  19 
respondents  (35.2%)  rated  risk  as  Moderate,  7  respondents 
(12.9%)  rated  risk  as  Low,  and  no  respondents  reported  that 
the  risk  associated  with  performance  was  Very  Low. 

B.  ffhat  causes  the  performance  risk?  First,  the 
Friedman  test  is  used  to  test  whether  the  performance  risk- 
driver  candidates  are  all  equally  weighted.  If  the  null 
hypothesis  fails  to  be  accepted,  then  the  pairwise  multiple 
comparison  will  determine  which  risk  drivers  are 
significantly  different  from  the  others.  The  original 
ranked  data  and  tied  data  are  shown  in  appendices  D  and  E. 
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The  test  procedure  and  results  are  shovm  in  tables  2,  3,  and 
4  below. 


Table  2 


Test  Hypothesis  for  Performance  Risk-Drivers 


1.  Null  hypothesis 
(H«) 

The  ten  risk-driver  candidates 
have  no  differential  effect. 

2.  Alternative 
hypothesis  (H^^) 

At  least  one  of  the  candidates 
tends  to  yield  larger  observed 
values  than  at  least  one  other 
candidate. 

3.  Test  statistics 

Where 

4.  Significance  level 

0.05 

5.  Critical  values 

where 

is  the  upper  a  percentile  for  a 
chi-square  distribution  with  k-1 
degrees  of  freedom. 

6.  Decision  rule 

Reject  the  null  hypothesis  if 

K  >. 

7.  Test  result 

K*»83.845  (7  respondents  rated 
the  risk  to  be  low,  and  there 
are  a  total  of  10  treatments, 
so,  n»47  k>10),  is  greater  than 
the  critical  value  16.919; 
therefore,  the  null  hvDothesis 
is  rejected. 
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Table  3 


Statistical  Factors  for  Multiple  Comparisons  for  Performance 

Risk-Drivers 


1  1.  Overall  error  rate 

0^0.45.  (10  treatments,  k(k- 
1 )/2-45  pairs ) . 

I2.  Significance  level 

1  for  two-treatment 

a‘=2*0.45/(10*9)=0.01.  From  Z 
table,  Zjji/2“2.575 

1  comparison 

1  3.  Significant 

1  difference  interval 

1*. ”***-* 

IR  j-Rj 1 >“2.575*29.35, >-75. 5 

In  table  3,  a  significant  difference  interval  of 
75.5  was  found,  which  means  that  any  two  treatments  with  R  j 
values  of  more  than  75.5  apart  represent  significantly 
different  treatments. 


Table  4 


Test  Result  of  Multiple  C<»parisons  for  Performance  Risk- 

Drivers 


Risk-driver 

candidate 

Description 

7 

Tight  schedule  for  program 
implementation. 

186.0 

1 

Requirements  are  not  clearly  and 
reasonably  stated. 

197.5 

5 

Technical  problems  cannot  be 
resolved  in  a  timely  manner. 

200.0 

8 

Poor  program  management  in  CSIST  or 
ROCAF,  or  both. 

206.5 
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Test  Result  of  Multiple  Cc»^arisons  for  Performance  Risk- 

Drivers  (Cont) 


6  Insufficient  manpower  to  carry  out  230.5 

the  tasks. 


2  Systems  designed  to  an  incomplete  271.0 

set  of  requirements. 


Developmental  testing  performed  272.5 

independently  by  contractor  and/or 
the  TEMP  implemented  inappropriately 
or  partially. _ 


3  Inadequate  configuration  control.  287.5 


9  Quality  assurance  isn't  well  planned  337.0 

_ or  isn't  carried  out  independently. _ 


Others.  389.0 


For  the  performance  risk-drivers,  statistically  the 
following  conclusions  were  drawn: 

1.  Tight  schedule  for  program  implementation  (7) 
drives  risk  more  than  incomplete  system  design  (2), 
inadequate  configuration  control  (3),  poor  quality  assurance 
(9),  and  others  (10); 

2.  Ambiguous  requirements  (1),  technical  bottleneck 
(5),  and  poor  program  management  (8)  drive  risk 
significantly  than  inadequate  configuration  control  (3), 
poor  quality  assurance  (9),  and  others  (10); 

3.  No  other  differences  are  significant. 
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SttDDortabillty  Risk.  Investigative  Questions  2.  What 


is  the  magnitude  of  supportabi 1 ity  risk  in  systems 
acquisition?  What  causes  the  risk? 

A.  What  is  the  siagnitude  of  supportabi  1  ity  risk  in 
systems  acquisition?  53  respondents  rated  supportabi 1 ity 
risk.  The  svimmarized  raw  data  is  in  appendix  F,  and  the 
histogram  (Figure  11)  shows  the  magnitude  of  supportabi 1 ity 
risk . 


Figure  11.  Supportabi 1 ity  Risk 


There  are  4  out  of  the  53  respondents  (7.6%)  who  rated 
the  supportabi 1 ity  risk  in  the  current  acquisition 
environment  as  Very  High,  25  respondents  (47.2%)  as  High,  17 
respondents  (32.0%)  rated  risk  as  Moderate,  7  respondents 
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(13.2%)  as  Low,  and  no  respondents  rated  the  magnitude  of 
supportability  risk  as  Very  Low. 

B.  What  causes  the  supportability  risk?  The  original 
ranked  data  and  tied  data  are  shown  in  appendices  G  and  H, 
respectively.  The  test  results  are  shown  in  tables  5,  6, 
and  7  below. 


Table  5 


Test  Hypothesis  for  Supportability  Risk-Drivers 


1. 

(Ho 

Null  hypothesis 

) 

The  eleven  risk-driver 
candidates  have  no  differential 
effect . 

2.  Alternative 
hypothesis  (H.^) 

At  least  one  of  the  candidates 
tends  to  yield  larger  observed 
values  than  at  least  one  other 
candidate. 

3. 

Test  statistics 

Where 

4. 

Significance  level 

0.05 

5. 

Critical  values 

X^o(k-l)»lB.3,  where  *2  (j^-d  ig 
the  upper  a  percentile  tor  a 
chi-square  distribution  with  k-1 
degrees  of  freedom. 

6. 

Decision  rule 

Reject  the  null  hypothesis  if 

K  >-  x2„(k-l). 
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Test  Hypothesis  for  Su 

pportability  Risk-Drivers  (Cent) 

7.  Test  result 

K^*54.3  (7  respondents  r:xtcd  the 
risk  to  be  low,  and  there  are 
total  11  treatments,  so,  n=46 
k=ll),  is  greater  than  the 
critical  value  18.3;  therefore, 
the  null  hvDothesis  is  relected. 

Statistical  Factors 
Supportak 

Table  6 

for  Multiple  Coav>arisons  for 
>ility  Risk-Drivers 

1.  Overall  error  rate 

a=0.55.  (11  treatments,  k(k- 
l)/2=55  pairs). 

2.  Significance  level 
for  two-treatment 
comparison 

a’=2*0.55/(ll*10)=0.01.  From  Z 
table,  Zjjt  /2=2 . 575 

3.  Significant 
difference  interval 

(R  ^-R  i l>«2. 575*31. 82, >»81. 9 

In  table  6,  a  significant  difference  interval  of  81.9 
was  found,  which  means  that  any  two  treatments  with  R  ^ 
values  of  more  than  81.9  apart  represent  significantly 
different  treatments. 
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Table  7 


Test  Result  of  Multiple  Coiiq;>ar Isons  for  Suppor tabl  1  i ty  Risk- 


Drivers 


Risk-driver 

candidate 

Description 

8 

Inexperienced  and/or  untrained 
personnel . 

186.0 

9 

Tight  schedule  and  insufficient 
manpower . 

233.0 

3 

Manpower  and  skill  analyses  are  based 
upon  experience  from  previous  system. 

244.5 

6 

Spares  are  provisioned  during  the 
development  phase (resulting  in 
incompatibility  with  field 
requirements),  or  initial  spares 
procured  by  part  number  without 
reference  to  the  specification. 

247.5 

10 

Poor  program  management  in  the  CSIST 
or  ROCAF,  or  both. 

263.5 

2 

Logistics  related  design  parameters 
are  established  after  other 
performance  parameters. 

265.5 

5 

Training  materials  and  equipment  are 
developed  based  on  initial  training 
used  for  T&E  personnel;  which  are  not 
adequate  for  field  use  due  to 
different  skill  levels,  etc. 

266.5 

4 

Requirements  are  not  clearly  and 
reasonably  stated. 

281.5 

7 

Technical  manuals  are  written  using 
the  EMD  LSA  results  which  doesn't 
reflect  the  production  configuration. 

323.0 

1 

Contractor  is  given  responsibility  to 
develop  the  LSA  program. 

325.0 

11 


Others . 


394.5 


For  the  supportability  risk-drivers,  the  following 
conclusions  were  drawn: 

1.  Inexperienced  and/or  untrained  personnel  (8)  drives 
risk  more  than  LSA  developed  by  contractor  (1),  ambiguous 
requirements  (4),  unrealistic  references  to  compile 
technical  manuals  (7),  and  others  (11); 

2.  Tight  schedule  and  insufficient  manpower  (9)  drives 
risk  more  than  LSA  developed  by  contractor  (1),  unrealistic 
references  to  compile  technical  manuals  (7),  and  others 
(11); 

3.  No  other  differences  are  significant. 

Cost  Risk.  Investigative  Question  3.  What  is  the 
magnitude  of  life-cycle  cost  risk  in  systems  acquisition? 
What  causes  the  risk? 

A.  What  is  the  magnitude  of  life-cycle  cost  risk  in 
systems  acquisition?  54  respondents  rated  life-cycle  cost 
risk.  The  summarized  raw  data  can  be  seen  in  appendix  I, 
and  the  following  histogram  (Figure  12)  shows  the  magnitude 
of  life-cycle  cost  risk. 
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Figure  12.  Life-Cycle  Cost  Risk 


There  are  7  out  of  the  54  respondents  (12.9%)  who 
rated  the  life-cycle  cost  risk  in  the  current  acquisition 
environment  as  Very  High,  28  respondents  (51.8%)  rated  it 
High,  15  respondents  (27.8%)  rated  it  Moderate,  4 
respondents  (7.5%)  as  Low,  and  no  respondents  rated  the 
magnitude  of  life-cycle  cost  risk  as  Very  Low.  Notice  that 
nearly  65%  of  the  respondents  rated  life-cycle  cost  risk  as 
High  or  Very  High. 

B.  What  causes  the  life-cycle  cost  risk?  The  original 
ranked  data  and  tied  data  are  shown  in  appendices  J  and  K, 
respectively.  The  test  results  are  shown  in  tables  8,  9, 
and  10  below. 
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Table  8 


Test  Hypothesis  for  Life-Cycle  Cost  Rlsk-Drlvere 


1.  Null  hypothesis 
(H«) 

The  eight  risk-driver  candidates 
have  no  differential  effect. 

2.  Alternative 
hypothesis  (H^^) 

At  least  one  of  the  candidates 
tends  to  yield  larger  observed 
values  than  at  least  one  other 
candidate. 

3.  Test  statistics 

Where 

4.  Significance  level 

0.05 

5.  Critical  values 

X^flj(k-1  )*14 . 06,  where 
is  the  upper  a  percentile  for  a 
chi-square  distribution  with  k-1 
degrees  of  freedom. 

6.  Decision  rule 

Reject  the  null  hypothesis  if 

K  >=  x2„(k-l). 

7.  Test  result 

K*3:78.98  (4  respondents  rated 
the  risk  to  be  low,  and  there 
are  total  8  treatments,  so,  n=50 
k«8),  is  greater  than  the 
critical  value  14.06;  so,  the 
null  hvDothesis  is  rejected. 
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Table  9 


Statistical  Factors  for  Multiple  C<»q;>arisons  for  Life-Cycle 


Cost 

i:  Risk-Drivers 

1.  Overall  error  rate 

a-0.28.  (8  treatments,  k(k- 
l)/2*28  pairs). 

2.  Significance  level 
for  two-treatment 
comparison 

o’=2*0.28/(8»7)=0.01.  From  Z 
table,  Zjjt/2“2.575 

3.  Significant 
difference  interval 

IR  ^-R  i l>=2.575*24.49,>-63.07 

In  table  9,  a  significant  difference  interval  of  63.07 
was  found,  which  means  that  any  two  treatments  with  R  j 
values  of  more  than  63.07  apart  represent  significantly 
different  treatments. 


Table  10 


Test  Result  of  Multiple  C<»v>arisons  for  Life-Cycle  Cost 

Risk-Drivers 


Risk-driver 

candidate 

Description 

1 

Ambiguous  requirements  or  frequent 
requirement  changes  which  cause 
major  engineering  changes  and/or 
system  redesigns. 

143.5 

5 

Sole-source  with  no  incentive  to 
control  cost  overrun. 

185.5 

3 

New  technology  is  introduced  without 
trade  studies  being  fully  conducted. 

213.0 
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Test  Result  of  Multiple  Cc»q;>arisons  for  Life-Cycle  Cost 

Risk-Drivers  (Cont) 


2 

Inadequate  logistics  related 
programming  which  results  in  a  high 
maintenance  burden. 

213.5 

6 

Tight  schedule  and/or  insufficient 
manpower . 

219.0 

7 

Improper  budget  control  system  which 
result  in  a  poor  understanding  of 
cost  distribution. 

232.5 

4 

Overall  test  prograun  is  not 
implemented  appropriately,  which 
results  in  rework  or  an  Increase  in 
maintenance  burden. 

247.5 

8 

Others . 

346.0 

For  the  life-cycle  cost  risk-drivers,  the 
following  conclusions  were  drawn: 

1.  Ambiguous  requirements  or  frequent  requirement 
changes  (1)  drives  risk  significantly  more  than  others 
drives  except  the  sole  source  environment  (5); 

2.  No  other  differences  are  significant. 

Schedule  Risk.  Investigative  Question  4.  What  is  the 
magnitude  of  schedule  risk  in  systems  acquisition?  What 
causes  the  risk? 

A.  What  is  the  magnitude  of  schedule  risk  in  systons 
acquisition?  53  respondents  rated  schedule  risk.  The 
summarized  raw  data  can  be  seen  in  appendix  L,  and  the 
following  histograun  (Figure  13)  shows  the  magnitude  of 
schedule  risk. 
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Figure  13.  Schedule  Risk 

There  are  6  out  of  the  53  respondents  (11.3%)  who  rated 
the  schedule  risk  in  the  current  acquisition  environment  as 
Very  High,  30  respondents  (56.7%)  as  High,  12  respondents 
(22.6%)  rated  schedule  risk  Moderate,  5  respondents  (9.4%) 
Low,  and  no  respondents  rated  the  magnitude  of  schedule  risk 
as  Very  Low.  Notice  that  68%  of  respondents  rated  schedule 
risk  as  High  or  Very  High. 

B.  Nhat  causes  the  schedule  risk?  The  original  ranked 
data  and  tied  data  are  shown  in  appendices  M  and  N, 
respectively.  The  test  results  are  shown  in  tables  11,  12, 
and  13. 
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Table  11 


Test  Hypothesis  for  Schedule  gisk-Drivers 


1 

1.  Null  hypothesis 
(H,) 

1 

The  nine  risk-driver  candidates 
have  no  differential  effect. 

2.  Alternative 
hypothesis  (H^) 

At  least  one  of  the  candidates 
tends  to  yield  larger  observed 
values  than  at  least  one  other 
candidate. 

3.  Test  statistics 

Where 

4.  Significance 
level 

0.05 

5.  Critical  values 

x2^(k-l)=15.5,  where  x^g(Jc-l)  is 
the  upper  a  percentile  for  a 
chi-square  distribution  with  k-1 
degrees  of  freedom. 

6.  Decision  rule 

Reject  the  null  hypothesis  if 

K  >=  x2„(k-l). 

7.  Test  result 

k’*-62.76  (5  respondents  rated 
the  risk  to  be  low,  and  there 
are  total  9  treatments,  so,  n«48 
k^’O),  is  greater  than  the 
critical  value  15.5;  so,  the 
null  hvDothesis  is  relected. 
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Table  12 


Statistical  Factors  for  Multiple  Coaq;>arisons  for  Schedule 

Risk-Drivers.  _ 


1.  Overall  error  rate 

a=0.36.  (9  treatments,  k(k- 
11/2=36  pairs) . 

2.  Significance  level 
for  two- treatment 
comparison 

a*=2*0. 36/(9*81=0.01.  From  Z 
table,  Zjji/2“2.575 

3.  Significant 
difference  interval 

1  "*'**^> 

IR  ^-R  , l>=2.575*26.83,>=69.09 

In  table  12,  a  significant  difference  interval  of  69.09 
was  found,  which  means  that  any  two  treatments  with  R  j 
values  of  more  than  69.09  apart  represent  significantly 
different  treatments. 


Table  13 


Test  Result  of  Multiple  Collar i sons  for  Schedule  Risk- 

Drivers 


Risk-driver 

candidate 

Description 

«.j 

1 

Ambiguous  requirements  or  frequent 
requirement  changes  which  cause 
major  engineering  changes  and/or 
system  redesigns. 

181.5 

8 

Late  involvement  of  production  and 
manufacturing  engineering. 

194.0 

3 

Poor  subcontractor  management. 

205.5 
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Test  Result  of  Multiple  C<»q;>arisons  for  Schedule  Risk- 

Drivers  (Cont) 


7 

Tight  schedule  and  insufficient 
manpower . 

217.5 

5 

New  technology  is  enforced  without 
trade  studies  being  fully  conducted. 

231.5 

6 

Overall  test  progreun  is  not 
implemented  adequately  or  timely 
which  results  in  system  rework. 

241.5 

2 

Poor  configuration  control. 

254.5 

4 

Work  Breakdown  Structure  has  not 
been  used  to  control  individual 
projects  and  progress  of  system 
integration. 

277.5 

9 

Others . 

357.0 

For  the  schedule  risk-drivers,  the  following 
conclusions  were  drawn: 

1.  Ambiguous  requirements  or  frequent  requirement 
changes  (1)  drives  risk  significantly  than  poor 
configuration  control  (2),  absence  of  using  WBS  to  control 
projects  (4)  and  others  (9); 

2.  Late  involvement  of  production  and  manufacturing 
engineering  (8)  and  Poor  subcontractor  management  (3)  drive 
schedule  risk  significantly  than  absence  of  using  HBS  to 
control  projects  (4)  and  others  (9); 

3.  No  other  differences  are  significant. 

Management  Risk.  Investigative  Question  5.  What  is 

the  magnitude  of  management  risk  in  systems  acquisition? 
What  causes  the  risk? 
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A.  What  is  the  magnitude  of  management  risk  in  systems 
acquisition?  54  respondents  rated  management  risk.  The 
summarized  raw  data  can  be  seen  in  appendix  0,  and  the 
following  histogram  (Figure  14)  presents  the  magnitude  of 
the  management  risk. 


Figure  14.  Management  Risk 


There  are  5  out  of  the  54  respondents  (9.3%)  who  rated 
the  management  risk  in  the  current  acquisition  environment 
as  Very  High,  19  respondents  (35.2%)  rated  it  High,  26 
respondents  (48.1%)  indicated  the  magnitude  of  management 
risk  was  Moderate,  4  respondents  (7.4%)  rated  it  Low,  and  no 
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respondents  indicated  the  magnitude  of  management  risk  was 
Very  Low. 

B.  What  causes  the  laanagMaent  risk?  The  original 
ranked  data  and  tied  data  are  shown  in  appendices  P  and  Q, 
respectively.  The  test  results  are  shown  in  tables  14,  15, 
and  16. 


Table  14 


Test  Hypothesis  for  Managwnent  Risk-Drivers 


1.  Null  hypothesis 
(H,,) 

The  eight  risk-driver  candidates 
have  no  differential  effect. 

2.  Alternative 
hypothesis  (H^) 

At  least  one  of  the  candidates 
tends  to  yield  larger  observed 
values  than  at  least  one  other 
candidate. 

3.  Test  statistics 

Where 

4.  Significance  level 

0.05 

5.  Critical  values 

X^g^{k-1)*1A.06,  where 
is  the  upper  a  percentile  for  a 
chi-square  distribution  with  k-1 
degrees  of  freedom. 

6.  Decision  rule 

Reject -the  null  hypothesis  if 

K  >- 
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_^^^^^J[e8^Hypothe8i8_£orManag«jen^Ri8k-Driver8(Cont)^^ 

7.  Test  result  k’^*30.1  (4  respondents  rated  the 

risk  to  be  low,  and  there  are 
total  8  treatments,  so,  n^BO 
k'^S),  is  greater  than  the 

1  critical  value  14.06;  so.  the 

null  hypothesis  is  rejected. 


Table  15 


Statistical  Factors  for  Multiple  Comparisons  for  Nanagestent 

Risk-Drivers 


1.  Overall  error  rate 

a»0.28.  (8  treatments,  k(k- 
l)/2>28  pairs) . 

2.  Significance  level 

o'=2*0.28/(8*7)=0.01.  From  Z 

for  two-treatment 

table,  1 

comparison 

3.  Significant 

difference- interval 

IR  ^-R  , l>*2.575*24.49,>-63.07 

In  table  15,  a  significant  difference  interval  of  63.07 
was  found,  which  means  that  any  two  treatments  with  R  j 
values  of  more  than  63.07  apart  represent  significantly 
different  treatments. 
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Tabla  16 


Test  Result  of  Multiple  Ccunparisons  for  Managonent  Risk- 

Drivers 


Risk-driver 

candidate 

Description 

5 

CSIST  and  ROCAF  lack  mutual 
understanding  of  program  objectives. 

141.0 

2 

Ambiguous  responsibility  and 
authority  defined  eunong  MND,  ROCAF 
and  CSIST. 

173.5 

1 

Ambiguous  requirements  or  frequent 
requirement  changes  which  complicate 
program  management. 

189.0 

6 

Lack  of  relevant  management  skills 
and  knowledge. 

200.5 

3 

Risk  management  has  not  been 
conceived  and  integrated  well  into 
program  management  strategy. 

218.5 

4 

Organizational  problems  such  as 
providing  limited  career  progression 
and  promotion  opportunities.  ROCAF 
or/and  CSIST  suffering  "brain-drain" 
problem. 

266.5 

7 

Organizational  problems  such  as 
bureaucracy,  lack  of /poor 
leadership,  power  struggles,  etc. 

274.5 

8 

Others . 

332.5 

For  the  Management  risk-drivers,  the  following 
conclusion  were  drawn: 

1.  Lack  of  relevant  management  skills  and  knowledge 
(1)  drives  management  risk  significantly  than  inappropriate 
risk  management  plan  (3),  organizational  problems — limited 
career  progression,  promotion  opportunities,  and  "brain- 
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drain"  (4),  another  organizational  problems  such  as 
bureaucracy,  lack  of/poor  leadership,  power  struggles  (7), 
and  others  (8); 

2.  Ambiguous  responsibility  and  authority  defined 
among  MND,  ROCAF  and  CSIST  is  ranked  significantly  different 
from  organizational  problems — limited  career  progression, 
promotion  opportunities,  and  "brain-drain"  (4),  another 
organizational  problems  such  as  bureaucracy,  lack  of/poor 
leadership,  power  struggles  (7),  and  others  (8); 

3.  No  other  differences  are  significant. 

Summary  of  The  Additional  Coronents 

Respondents  were  asked  to  comment  on  the  risk  factors 
and  their  causes  in  part  III  of  the  questionnaire  as  a  means 
of  collecting  additional  research  information.  The  comments 
are  summarized  and  categorized  by  specific  risk  factor. 

A.  Comments  on  Performance  Risk. 

1.  Systems  engineers  for  both  the  CSIST  and  RCX:af  (or 
services)  need  to  be  sufficiently  trained.  Systems 
integration  is  a  great  challenge  in  the  current  ROCAF 
acquisition  environment. 

2.  Inputs  for  Mission  Needs  Statements  and  Operational 
Requirements  Documents,  such  as,  the  overall  national 
defense  strategy,  technologies  and  industrial  capabilities, 
and  so  forth,  should  be  realistic  and  well  planned. 

3.  Adequate  strategy  for  concurrence  in  development 
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and  production,  when  it  is  considered  to  be  desirable, 
should  be  well  planned. 

4.  End  users  must  be  a  part  of  the  program  from  the 
very  beginning. 

5.  Communications  between  the  CSIST,  MND  and  ROCAF 
need  to  be  improved. 

B.  Comments  on  Supportabl 1 1 tv  Risk. 

1.  Definitions  for  logistics  requirements  such  as 
reliability,  maintainability,  systems  safety,  and  so  forth, 
need  to  be  well  stipulated,  and  the  requirements  should  be 
carefully  incorporated  into  systems  designs. 

2.  The  current  status  of  field  logistics  should  be 
considered  when  developing  the  integrated  logistics  plan. 

3.  Concurrence  in  development  and  production  results 
in  additional  uncertainty  for  reliability,  maintainability, 
and  systems  safety  which,  without  proper  care,  will  result 
in  increasing  supportabi 1 ity  risk  and  operation  and  support 
costs . 

4.  Communications  between  the  CSIST  and  ROCAF  in  terms 
of  logistics  requirements  allocation  need  to  improve  to 
minimize  risk  magnitude. 

C.  CCTunents  on  Life-Cycle  Cost  Risk. 

1.  Cost  risk  is  inevitable  due  to  the  lack  of  relevant 
experience  in  weapon  systems  designs,  development,  and 
production  by  the  CSIST. 
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2.  The  CSIST  has  few  incentives  to  control  program 
costs  since  they  are  a  sole-source  provider  and  a 
government -owned  organization.  Business-like  organizational 
changes  should  be  considered  for  the  CSIST. 

3.  Cost  control  is  hampered  by  a  lack  of  historical 
data  for  cost  estimation.  Therefore,  the  development  of 
weapon  systems  cost  information  database  is  highly 
recommended. 

D.  Comments  on  Schedule  Bisk. 

1.  Critical  systems  technologies  increase  schedule 
risk  if  such  technologies  are  not  under  the  auspices  of  the 
CSIST. 

2.  Potentially,  concurrence  in  development  and 
production  could  increase  the  number  of  engineering  changes, 
which  is  detrimental  to  schedule  control.  It  is  very 
important  to  prepare  for  the  impacts  of  concurrence  in 
advance  and  to  adopt  an  appropriate  acquisition  strategy. 

3.  Detailed  breakdowns  of  work  are  fundamental  for 
successfully  managing  schedule  and  cost. 

4.  Political  involvement  should  be  minimized  once  the 
milestone  decisions  are  made. 

E.  Comnents  on  Manaawaent  Risk. 

1.  Having  "birds-eye  view"  of  the  program  for  high- 
level  managers  is  critical  to  the  success  of  the  programs. 

2.  Subjective  and  objective  program  environments 
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should  be  considered  thoroughly  before  determining  the 
acquisition  strategy. 

3.  The  proper  tour  length  for  management  personnel 
should  be  well  defined. 

4.  The  organizational  culture,  including  the  promotion 
system,  leadership,  authority,  and  responsibility  need  to  be 
well  defined  in  order  to  alleviate  the  frustration  of 
engineers  and  managers. 

Summary  of  Findings 

Figure  15  illustrates  the  percentage  of  respondents  who 
rated  performance,  supportability,  life-cycle  cost, 
schedule,  and  management  high  or  very  high  risk.  Notice 
that  except  management  risk,  more  than  50%  of  respondents 
rated  all  other  risk  factors  high  or  very  high  risk,  in 
which  schedule  risk  appears  to  be  the  most  important  risk, 
because  nearly  70%  of  respondents  rated  it  high  or  very  high 
risk. 
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Figure  15.  Percentage  of  Respondents  Who  Rated  High  or 
Very  High  Risk  to  the  Five  Risk  Factors 

Tabie  17  illustrates  the  risk  magnitudes  and  drivers 
that  were  found  in  this  chapter.  Column  1  categorizes  the 
risk  factors;  column  2  presents  the  mode  responses  for  risk 
magnitude;  and  column  3  summarizes  the  most  important  risk 
drivers . 
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Risk  Factors 


Risk 

Magnitude 

High 


Performance 


Supportabi 1 ity 


High 


Cost 


High 


17 


ed  Findings 
Risk  Drivers 


.  Schedule  is  too  tight  to 
implement  required  tasks 
effectively. 

.  Requirements  are  not 
clearly  and  reasonably 
stated. 

.  Technical  bottlenecks 
exist,  and  are  not  exposed 
and  solved  in  a  timely 
manner . 

.  Poor  program  management  of 
the  CSIST  or  the  RCX^AF,  or 
both. _ 

.  Inexperienced  and/or 
untrained  personnel  are 
responsible  for 
logistics  planning  and 
programming. 

.  A  tight  schedule  and  a 
shortage  of  systems 
engineers  for  conducting 
the  required  logistics 
tasks. _ 

.  Ambiguous  requirements  or 
frequent  requirement 
changes  which  result  in 
major  engineering  changes 
and/or  redesigns. _ 


Smwaariaed  Findings  (Cont) 


Schedule 

High 

.  Ambiguous  requirements  or 
frequent  requirement 
changes  which  result  in 
major  engineering  changes 
and/or  redesigns. 

.  Ineffective  programming  of 
systems  integration 
resulting  in  major  schedule 
deviations  or  late 
involvement  of  production 
and  manufacturing 
engineering. 

.  Poor  subcontractor 
management . 

Management 

Moderate 

.  CSIST  and  ROCAF  lack  a 
mutual  understanding  of 
program  objectives,  which 
seriously  complicates 
program  management . 

.  Ambiguous  lines  of 
responsibility  and 
authority  defined  between 
the  MND,  ROCAF  and  CSIST. 

Recall  from  figure  8  that  the  magnitudes  of  the  risk 
factors  were  measured  by  the  respondents,  a  majority  of  whom 
declared  a  risk  propensity  of  risk  neutral  with  a  few  risk 
avoiders.  However,  the  risk-neutral  propensity  of  the 
respondents  reveals  that  the  rated  risk  magnitudes  for  those 
factors  are  probably  accurate.  It  is  also  possible  the  risk 
magnitudes  responded  in  this  study  are  slightly 
overestimated  since  37%  of  the  respondents  are  risk  averse. 

The  result  of  self-evaluation  of  management  competency 
by  the  respondents  also  warrant  discussion.  Nearly  89%  of 
respondents  indicated  they  are  competent  in  their  day-to-day 
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management,  and  do  not  request  any  additional  managerial 
training.  This  contrasts  with  results  of  the  survey 
conducted  at  Wright-Pat terson  Air  Force  Base  in  1989  where 
ASD  managers  requested  more  training  for  competency  (4:97). 
According  to  Edwards  W.  Deming,  the  American  who  taught  the 
Japanese  about  quality,  quality  is  the  responsibility  of 
management  (33:116-7),  and  in  fact  Deming  asserts  that  most 
troubles  and  most  possibilities  for  improvement  add  up  to 
proportions  of  94%  belongs  to  management  responsibility  (36 
:315).  In  this  study  the  perceived  ris]c  associated  with 
management  was  lower  than  all  other  categories.  Since  the 
survey  sample  included  only  mid-to-high  level  managers,  the 
results  of  this  study  suggests  the  managers  may  be 
underestimating  the  extent  to  which  management  is 
contributing  to  risic  in  ROCAF  system  acquisition  management. 

Program  managers  must  be  the  primary  agents  for 
improvement  (33:182).  In  fact,  management  should  assume 
responsibility  for  two  management  r isle-drivers  that  were 
identified.  First,  the  CSIST  and  R(X;af  both  laclc  a  mutual 
understandings  of  program  objectives.  Second,  ambiguous 
lines  of  responsibility  and  authority  defined  between  the 
MND,  ROCAF  and  CSIST. 

Sumnary 

This  chapter  has  examined  the  acquired  data  according 
to  the  methodologies  introduced  in  chapter  III,  and  has 
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achieved  the  research  objectives  stated  in  the  chapter  I. 

The  first  objective  was  to  explore  the  magnitude  of 
potential  major  risks  in  terms  of  performance,  cost, 
schedule,  management,  and  supportability  in  the  ROCAF's 
major  systems  acquisition  process.  The  second  objective  was 
to  identify  and  understand  risk-drivers  in  the  ROCAF  weapon 
systems  acquisition  process.  Chapter  V  presents  the 
conclusions  and  recommendations  of  the  research. 
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V.  Conclusions  and  ReccMMaendatlons 


Overview 

This  chapter  will  provide  the  conclusions  of  this 
research.  It  will  then  address  some  recommendations 
proposed  by  the  researcher  to  mitigate  the  risks  perceived 
by  the  respondents  in  the  ROCAF’s  weapon  systems 
acquisition.  Finally,  this  chapter  will  present  some 
recommendations  for  future  studies. 

Conclusions 

This  study  determined  that  the  risks  of  performance, 
supportabi 1 i ty ,  cost,  and  schedule  are  high,  and  management 
risk  is  moderate.  Risk  magnitude  refers  to  the  combination 
of  probability  of  occurrence  of  an  unwanted  outcome  and  the 
severity  of  impact.  High  risk  means  that  this  combination 
is  more  than  likely  to  happen.  Since  resources  are  limited, 
management  should  be  aware  of  risks  and  endeavor  to 
alleviate  the  associated  impacts.  Although  it  is  known  that 
cost,  schedule,  supportabi 1 i ty,  and  performance  risks  are 
often  related  and  result  from  the  same  risk  drivers,  two 
risk  drivers  actually  resulted  in  high  risk  for  more  than 
one  risk-factor  in  the  ROCAF's  acquisitions.  Management 
efforts  should  be  concentrated  on  these  two  drivers 
according y  The  multi-dimensional  drivers  are: 

A.  Ambiguous  requirements  or  frequent  requirement 
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changes  drive  performance,  life-cycle  cost,  and  schedule 
risks  of  the  ROCAF  acquisition. 

B.  Tight  program  schedules  caused  by  the  need  for 
rapid  systems  deployment  contribute  to  both  performance  and 
supportability  risks. 

Frequently  it  is  very  difficult  to  mitigate  risks  which 
result  from  external  factors.  This  is  the  case  with  the 
second  risk  driver-tight  schedules,  because  the  schedules 
are  typically  policy  driven. 

Analogy  to  U.S.  Acquisition  Problems 

Notwithstanding  the  nximerous  defense  programs  which 
have  been  completed  in  the  U.S.,  many  researchers  and 
presidential  commissions  have  concluded  repeatedly  that 
opportunities  exist  to  save  billions  of  dollars  by  improving 
the  acquisition  process.  The  studies  repeatedly  urged 
Congress  and  the  DoD  to  correct  five  basic  deficiencies 
which  are  similar  to  those  reported  in  chapter  IV,  namely: 

A.  Unrealistic  requirements  for  the  most  sophisticated 
systems  attainable,  often  irrespective  of  cost; 

B.  Underestimated  schedules  and  costs  of  major 
programs,  which  distort  the  decision-making  process  for  the 
allocation  of  the  national  budget; 

C.  Changes  in  program  and  contract  requirements  caused 
by  changes  in  military  user  preferences,  leading  to  annual 
or  more  frequent  changes  in  program  funding  levels. 
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initiated  by  either  Congress  or  the  DoD  itself; 


D.  Lack  of  incentives  for  contractors  and  government 
personnel  to  reduce  program  costs;  and 

E.  Failure  to  develop  sufficient  numbers  of  military 
and  civilian  personnel  with  training  and  experience  in 
business  management  and  in  dealing  with  industrial  firms  to 
oversee  the  development  and  production  of  enormous,  highly 
technical  industrial  programs  (35:32). 

Recommendations 

As  indicated  previously,  the  purpose  of  risk 
identification  is  to  mitigate  the  risks.  The  following 
recommendations  are  made  to  assist  the  ROCAF  acquisition 
process  and  are  based  on  comments  from  the  survey 
respondents.  The  recommendations  are  organized  according  to 
the  identified  risk  drivers  in  the  following  tables  (18-22). 


Table  18 


Recommendations  for  Mitigating  Perfonaance  Risk 


Identified  Risk 

Drivers 

RecosBMndat  ions 

A.  Tight  schedule  to 
implement  required 
tasks  effectively. 

A.l.  Defense  discrepancies  and 
mission  needs  should  be 
identified  accurately  in  a 
timely  manner.  The  resolution 
of  this  problem  requires 
constant  review  efforts. 

A. 2.  The  acquisition  strategy 
should  address  schedule 
constraints.  Concurrent 
engineering  is  recommended. 

Recornnendations  for  Mitigating  Performance  Risk  (Cont) 


Identified  Risk 

Drivers 

Recomnendat i ons 

B.  Requirements  are 

B.l.  Employ  a  consultant 

not  clearly  and 

company  to  help  prepare  clear 

reasonably  stated. 

requirements  documents  when  the 
in-house  capability  hasn't  yet 
been  established. 

B.2.  Communicate  with  the 

CSIST  to  assure  a  clear 
understanding  of  the 
requirements . 

B-3.  Comprehend  state-of-the- 
art  and  advanced  technologies 
would  help  define  requirements. 
B.4.  Establish  a  technology 
information  library. 

C.  Technical 

C.l.  The  Dem/Val  phase  of  the 

bottlenecks  exist  and 

program  should  be  retained 

aren't  identified  in  a 

since  it  helps  to  isolate 

timely  manner. 

technical  bottlenecks. 

C.2.  The  application  of 
concurrent  engineering  could 
detect  technical  pitfalls  in 
the  various  phases. 

D.  Poor  program 

D.  Appropriate  personnel 

management  in  the 

training  is  required.  The 

CSIST,  or  the  ROCAF, 

National  Defense  Management 

or  both. 

College  should  expand  its 
educational  programs  or  offer 
professional  continuing 
education  for  systems 
management . 

Table  19 


Reconaiendatlons  for  Mitigating  Supportabllity  Risk 


Identified  Risk 

Drivers 

Recomendat  ions 

A.  Inexperienced 
and/or  untrained 
personnel  are 
responsible  for 
logistics  planning  and 
programming. 

A.l.  Employ  a  consultant 
company  to  support  the 
logistics  planning  and 
programming. 

A. 2.  Appropriate  personnel 
training  is  required.  The 
National  Defense  Management 
College  should  expand  its 
educational  programs  or  offer 
the  professional  continuing 
education  for  systems 
management . 

A. 3.  In  addition  to  the 
traditional  training  programs, 
coaching  and  mentoring  are  also 
good  ways  of  developing 
competent  personnel . 

B.  Tight  schedules 
and  a  shortage  of 
systems  engineers  for 
conducting  the 
required  logistics 
tasks . 

B.l.  The  acquisition  strategy 
should  address  schedule 
constraints.  Concurrent 
engineering  is  recommended. 

B.2.  Manpower  shortages  can  be 
resolved  in  two  ways: 

B.2.a.  Systems  engineers  for 
logistics  should  be  trained. 
B.2.b.  To  minimize  the  "brain- 
drain"  problem  in  both  the 

CSIST  and  ROCAF,  the 
organizational  cultures 
including  the  promotion  system, 
leadership,  authority,  and 
responsibility;  need  to  be  well 
defined  and  cultivated  in  order 
to  minimize  the  frustration  of 
engineers  and  managers. 

99 


Table  20 


ReconwiendationB  for  Mitigating  Life-Cycle  Cost  Risk 


Identified  Risk 

Drivers 

Recomnendat ions 

A.  Ambiguous 
requirements  or 
frequent  requirement 
changes  which  result 
in  major  engineering 
changes  and/or 
redesigns . 

A.l.  To  resolve  the  ambiguous 
requirements,  the  ROCAF  can: 

A.l. a.  Use  a  consultant 
company  to  help  define  clear 
requirements . 

A.l.b.  Communicate  with  the 

CSIST  to  assure  a  clear 
understanding  of  the 
requirements . 

A. 2.  It  is  important  to 
understand  that  requirements 
will  change  to  coincide  with 
changes  of  the  threats. 
Nonetheless,  in  order  to 
alleviate  the  impacts  of  the 
changes,  systems  design  should 
consider  appropriate 
flexibility  for  requirements 
changes . 

A. 3.  Concurrent  engineering 
can  be  used  to  determine  and 
minimize  the  impacts  of 
requirement  changes. 

Table  21 

Recownendations  for  Mitigating  Schedule  Rlak 


Identified  Risk 
Drivers 


A.  Ambiguous 
requirements  or 
frequent  requirement 
changes  which  result 
in  major  engineering 
changes  and/or 
redesigns^ _ 


Recomaenda t ions 


A.  See  the  recommendations 
Life-Cycle  Cost  Risk. 


for 
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Recommendations  for  Mitigating  Schedule  Risk  (Cont) 


Identified  Risk 

Drivers 

Recommendat ions 

B.  Ineffective 
systems  integration 
which  causes 
complicated  schedule 
control  or  late 
involvement  of 
production  and 
manufacturing 
engineering  which 
increases  schedule 
risk. 

B.l.  Good  systems  integration 
calls  for  well  trained  systems 
engineers.  More  systems 
engineers  should  be  trained. 

B.2.  Create  motivation  to 
minimize  the  "brain  drain"  from 
the  CSIST  and  ROCAF. 

B.3.  A  consultant  company  can 
be  used  to  help  systems 
integration. 

B.4.  Concurrent  engineering 
can  be  adopted  to  introduce 
early  involvement  of  production 
and  manufacturing  engineering. 

C.  Poor  subcontractor 
management . 

C.l.  Statements  of  Work  should 
be  clearly  defined  to  stipulate 
all  non-specification 
requirements  for  contractor 
efforts  either  directly  or  with 
the  use  of  specific  cited 
documents . 

C.2.  Many  techniques  such  as 
Cost/Schedule  Control  System 
Criteria  (C/SCSC),  Technical 
Performance  Measurement  (TPM), 
and  Program  Evaluation  Review 
Technique  (PERT)  can  be  used. 
According  to  the 
questionnaires,  these 
techniques  are  not  currently 
being  used  extensively  with  the 
exception  of  Work  Breakdown 
Structures  (WBS). 
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Table  22 

^^^^^^^HecOTmemdaMone^forjtitioatlngManaoementRisk^^ 

Identified  Risk  RecomBendations 

Drivers _ 

A.  Lack  of  mutual  A.l.  Make  the  defense  programs 

understanding  on  more  business-like  to  build 

program  objectives  and  mutual  understanding  since,  by 

management  between  the  definition,  the  current 

CSIST  and  ROCAF.  differences  between  the 

organizational  goals  introduce 
conflicts  in  program 
management . 

A. 2.  More  detailed  defense 
acquisition  regulations 
defining  responsibilities, 
authorities,  and 
accountabilities  should  be 
instituted. 


B.  Ambiguous  lines  of  B.  Same  as  for  A. 

responsibility  and 

authority  defined 

between  the  MND,  ROCAF 

and  CSIST.  Currently, 

this  ambiguity  causes 

severe  impacts  to  the 

programs  and  their 

management . 


For  day-to-day  program  management,  the  DSMC  suggests 


that  the  application  of  the  "Transition  from  Development  to 
Production"  templates  and  the  "Best  Practices"  guide  should 
reduce  many  program  risks.  The  risk  drivers  are  almost 
identical  to  the  "traps"  and  "areas  of  risk"  suggested  by 
these  publications,  hence  both  could  prove  to  be  invaluable 
tools  to  ROCAF  weapons  acquisition  management. 
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A  careful  review  of  the  recoimnendatione  stated 
previously  shows  that  increasing  systems-engineering 
manpower  and  applying  concurrent  engineering  are  important 
strategies  to  confront  the  diverse  performance, 
supportabi 1 ity ,  life-cycle  cost,  and  schedule  risks  for  the 
following  reasons: 

A.  Systems  engineers  for  both  the  CSIST  and  RCXTAF 
should  be  trained  as  suggested  in  this  study.  These 
engineers  can  help  to  analy2e,  translate  and  allocate 
requirements  into  specifications  as  well  as  anticipate 
program  traps  in  terms  of  technology  insertion  and  systems 
integration. 

B.  According  to  the  comments  from  the  respondents,  an 
adequate  strategy  for  managing  concurrency  of  phases  due  to 
the  demands  of  faster  product  delivery  should  be  well 
planned.  This  problem  cannot  be  overcome  without  competent 
systems  engineers. 

C.  Concurrent  engineering  is  defined  as 

a  systematic  approach  to  the  integrated,  concurrent 
design  of  products  and  their  related  processes, 
including  manufacturing  and  support.  This  approach  is 
intended  to  cause  the  developers,  frcsn  the  onset,  to 
consider  all  elmnents  of  the  product  life  cycle  from 
conception  through  disposal,  including  quality,  cost, 
schedule  and  user  requireaients  (29:18). 

Concurrent  engineering  is  a  relatively  recent  time-based 

management  innovation  directed  to  shortening  the  product  or 

service  development  (32:7)  and  might  enhance  product  quality 
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if  the  challenges  of  its  application  can  be  overcome. 

The  framework  for  application  of  concurrent  engineering 
is  shown  in  figure  16. 


Figure  16.  Framework  for  Application  of  Concurrent 


Engineering  (29:19) 

The  challenges  of  implementing  concurrent  engineering, 
according  to  Program  Manager,  a  DSMC  publication,  were: 
organizational  structures,  business  practices,  funding  and 
budgeting  for  concurrent  engineering,  education,  integration 
of  cost,  schedule,  performance  and  risk  (29:20). 

Ten  problems  related  to  overcoming  these  challenges 
were  identified  by  the  DoD  in  1990  in  which  lack  of 
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universal  understanding  of  the  philosophy,  processes,  tools 
and  practices  of  concurrent  engineering  was  the  major 
problem  (29:20).  Five  solutions  were  suggested  by  the  DoD 
concurrent  engineering  workshop,  which  also  can  be  used  to 
the  ROCAF  and  other  services  acquisition  management: 

A.  Top  management  support  for  concurrent  engineering 
education. 

B.  Educate  the  DoD  and  industry  infrastructures  in 
concurrent  engineering  education. 

C.  Establish  concurrent  engineering  education 
opportunities. 

D.  Design  curricula  for  concurrent  engineering. 

E.  Dialogue  within  and  among  constituencies  on 
cone,  ent  engineering  (29:22-3). 

Notice  that  the  accomplishment  of  concurrent 
engineering  calls  for  Integration  of  diverse  disciplines. 
Normally  systems  engineers  will  act  as  integrators  to  assure 
the  success  of  its  application.  This  further  highlights  the 
importance  of  the  first  recommendation  stated  previously, 
i.e,  systems  engineers  for  both  the  CSIST  and  ROCAF  should 
be  trained  sufficiently.  Establishment  of  this  training 
program  should  be  given  precedence  in  order  to  minimize 
risks  in  ROCAF 's  weapon  systems  acquisitions. 
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Recomnendations  for  Future  Studies 


Since  this  research  was  a  pilot  study  of  ROCAF  weapon 
systems  acquisition  risk,  its  scope  of  research  is  limited 
to  risk  identification.  Since  risk  identification  is  only  a 
part  of  risk  management  which  ultimately  requires  risk 
handling,  there  are  many  areas  that  have  not  been  covered  in 
this  research  which  should  be  studied,  such  as: 

A.  This  study  did  not  identify  that  CSIST,  a 
government -owned  and  nonprofit  organization,  was  a 
statistically  significant  cost  risk  driver.  However,  the 
status  of  the  CSIST  shows  that  it  is  still  an  important 
driver.  Technical  study  should  be  conducted  to  investigate 
how  a  government -owned  or  nonprofit  organization  can  be 
incentivized  to  control  program  expenses.  When  this  study 
is  completed,  it  can  be  used  as  a  reference  for  considering 
the  organizational  changes  of  such  organizations. 

B.  Since  resource  constraints,  such  as  the  manpower 
shortage  in  the  Weapon  Systems  Acquisition  Management  Office 
(WSAMO) ,  a  newly-activated  office  in  the  ROCAF  general 
headquarters,  determine  the  likelihood  of  inherent  risks  in 
weapon  acquisitions,  research  should  be  conducted  to  analyze 
cost/benefit  trade-offs  study  associated  with  integrating 
the  ROCAF  and  CSIST  manpower  and  centralizing  program 
management . 

C.  Acceptable  risk  has  been  reviewed  briefly  in 
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chapter  II  and  will  vary  on  a  case  by  case  basis.  However, 
for  each  acquisition  activity  in  which  the  subjective  and 
objective  acquisition  environments  are  different,  it  would 
be  useful  to  conduct  a  study  quidinq  overall  considerations 
for  acceptable  risk  of  the  ROCAF  and  other  services' 
acquisition  environment. 
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Appendix  A;  Questionnaire  for  Risk  Identification  in  Malor  Weapon 
Systems  Acquisition  in  the  ROCAF  (English) 

Purpose 

To  identify  the  risk  areas  and  their  drivers  in  major  weapon 
systems  acquisition  in  the  ROCAF  from  which  a  universal 
understanding  of  effective  program  management  can  be  made. 

Justification 

For  the  defense  programs  that  are  undergoing  certain 
constraints  in  the  ROCAF,  it's  desired  to  understand  the 
subjective  and  objective  environments  and  how  those 
environments,  or  constraints,  affect  the  weapon  systems 
acquisition  prior  to  program  implementation.  From  the 
study,  an  awareness  of  the  likelihood  of  occurrence  of 
unwanted  outcomes  of  programs  can  be  determined  and  remedial 
actions  can  be  taken  accordingly.  It  is  important  to 
emphasize  that  the  purpose  of  this  survey  is  not  to  review 
particular  defense  projects,  but  rather  to  learn  the 
constraints  and  the  means  for  dealing  with  them. 

Method 

Risk  Assessment  is  dependent  upon  one's  insight.  The 
Expert  Interview  (survey)  is  considered  the  most  appropriate 
of  the  methods  suggested  by  the  U.S.  Defense  System 
Management  College  in  terms  of  reaching  the  research 
objectives  as  well  as  the  goal  stated  in  the  previous 
paragraph. 


108 


Sampl ing 


Since  the  population  of  defense  program  managers  is 
relatively  small,  and  also  since  the  IDF  program  receives  a 
great  deal  of  the  defense  budget  and  manpower,  those  mid-to- 
high  level  managers  in  the  IDF  program,  in  both  the  ROCAF 
and  CSIST,  are  considered  to  be  the  most  capable  program 
managers  currently  in  the  defense  acquisition  process.  To 
survey  those  managers  is,  therefore,  considered  an  unbiased 
estimation  technique  for  the  causation  and  magnitudes  of  the 
acquisition  risks. 

Anonymity 

The  disclosure  of  your  name  and/or  position  is 
absolutely  optional.  In  order  to  secure  the  content 
validity  of  the  research,  each  questionnaire  will  be  sealed 
in  an  envelop  and,  the  confidentiality  of  your  responses  is 
guaranteed.  If  you  choose  to  print  your  name  and/or 
position  this  information  will  be  considered  when  the  author 
conducts  the  statistical  analysis.  However,  weighting  is 
also  based  upon  the  answer  given  during  the  self-appraisal 
question  in  the  first  questionnaire  and  upon  the  author's 
discretion. 

Response  Guide 

A.  The  questionnaire  is  segmented  into  three  parts  for 
each  question.  You  will  be  asked  to  answer  at  least  the 
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first  two  ACCORDING  TO  YOUR  OWN  OPINIONS.  The  first  part 
concerns  the  magnitude  of  the  performance,  supportabi 1 ity , 
cost,  schedule  and  management  risk.  The  second  part 
concerns  risk  causation.  You  are  also  invited  to  comment  on 
any  other  aspects  of  weapon  systems  acquisition. 

B.  Before  choosing  your  answer,  it's  recommended  that 
you  preview  the  question  and  all  possible  choices. 

C.  For  the  first  part  of  the  question,  choose  only  one 
answer  from  the  list. 

D.  For  the  second  part,  either  choose  and  prioritize 
your  answer  from  the  list  provided  or  write-in  your  own 
answer  in  the  space  provided. 

E.  Magnitude  of  risk  is  somewhat  qualitative  and 
subjective; 
however ,  i t  1 s 
requested  that  you 
attempt  to 
quantify  the 
magnitude  of  the 
risk  in  terms  of 
probability  amount 
(that  is,  1%,  34%, 

50% ,  68% , . .  . , 
etc.)  before 

selecting  a  "preference"  for  that  particular  question.  The 
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figure'  provides  an  overall  picture  for  rating  the  magnitude 
of  risk.  For  example,  a  70%  probability  of  occurrence 
accompanied  with  5%  severity  of  impact  (determined  by  you) 
would  be  considered  low  risk,  whereas  20%  probability  of 
occurrence  with  40%  severity  of  impact  would  be  considered 
moderate  risk. 


Risk  Management  Concept  and  Guidance t 4-9 
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Questionnaire  about  Yourself 


A.  Name  (optional) 


Feb  10  1992 


B.  Service  and  Position  (optional) 


C.  Years  in  the  program  (to  one  decimal  place) 


D.  Is  your  current  job  mostly  managerial  or  technical 
(checlc  one) 

M -  E - 

E.  Familiarity  of  weapon  systems  acquisition  process  and 
its  management  (check  one) 


1.  Highly  competent  (If  you  feel  your  day-to-day 

-  management  and  decision-making 

are  very  easy) 

2.  Competent  (If  you  feel  your  day-to-day 

-  management  and  decision¬ 
making  are  easy  and  give 
you  very  few  problems.) 

3.  Moderate  (If  you  feel  your  day-to 

-  (Jay  management  and  decision¬ 
making  are  somewhat  difficult, 
but  everyone  has  the  same 
management  problem. ) 

4.  More  improvement  needed  (If  you  feel  your 

- day-to-day  management 

and  decision-making  are 
difficult  and  cause  you 
problems,  but  that 
additional  training 
would  improve  the 
situation) 


5.  Unqualified  (If  you  feel  your  day- 

- to-day  management  and 

decision-making  is 
very  difficult  and  that  no 
amount  of  additional  training 
would  improve  the  situation) 


(continued  on  next  page) 


112 


F.  Concerning  your  personal  propensity  for  risk  taking 
in  your  day-to-day  life,  are  you 

A  Risk  Taker  Risk  Neutral  A  Risk  Avoider 


(Example:  you  might  rate  yourself  as  a  risk  taker  if 
you  always  speculate  in  stocks.) 


G.  Are  you  aware  of  any  risk  management  techniques 
available  to  you  (such  as  the  Work  Breakdown  Structure, 
Cost/Schedule  Control  System  Criteria  (C/SCSC),  Technical 
Performance  Measurement  (TPM),  Venture  Evaluation  Review 
Technique  (VERT),  Program  Evaluation  Review  Technique 
(PERT),  etc.) 


Yes  Somewhat  No 


(If  your  answer  for  question  G  is  Yes,  please 
continue  with  question  H,  otherwise  continue  to  the  next 
page) 

H.  Do  you  use  risk  management  techniques  in  your  program? 


Yes  Somewhat  No 
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Questionnaire  for  Performance  Risk  Identionf ication 


Part  1.  Based  on  your  experiences  and  understanding,  if 
current  acquisition  variables  (resource  availability, 
policies, etc. )  remain  unchanged,  how  will  it  affect 
performance  risk  in  major  defense  systems? 

(Definition  of  performance  risk:  Likelihood  that  defined 
operational  effectiveness  such  as  maneuverability, 
durability,  and  Pilot  Vehicle  Interface  will  not  be 
achieved.  Note,  these  do  NOT  include  suitability.) 

(Please  check  one) 

Very  high  High  Moderate  Low  Very  Low 


Part  2.  If  your  answer  in  Part  1  was  Moderate,  High,  or 
Very  High,  please  rank  the  candidate(s)  which  is  (are)  the 
likely  risk  driver(s)  from  the  list  below.  You  may  check  as 
many  candidates  as  you  wish  using  the  numbers  1,  2,  3,  4,  5, 
6,  7,  8,  9  where  1  represents  the  most  critical  driver  and  9 
represents  the  least  critical  diver.  Do  not  answer  this 
question  if  your  answer  in  Part  1  was  Low  or  Very  Low. 

EXAMPLE:  (  3  )  1.************** 

(  2  )  2.************** 

(  )  1.  Requirements  are  not  clearly  and  reasonably 

stated' . 

(  )  2.  Systems  design  analysis  is  not  carefully 

conducted.  In  other  words,  the  system  is  designed  to  an 
incomplete  set  of  requirements 

(  )  3.  Bad  configuration  control. 

(  )  4.  Developmental  testing  is  independently  performed 

by  the  contractor  and/or  the  Test  and  Evaluation  Master  Plan 
is  not  implemented  appropriately  and  thoroughly. 

(  )  5.  Technical  problems  can  not  be  resolved  in  a 

timely  manner. 

(  )  6.  Insufficient  manpower  to  carry  out  the  tasks. 


'  Most  of  candidate  drivers  were  taken  from  the  Beet  Practices 
and  DoD  4245. 7-M — Transition  from  Development  to  Production. 
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(  )  7.  Tight  schedule  to  implement  the  tasks 

sufficiently. 

(  )  8.  Poor  program  management  in  CSIST  or  ROCAF,  or 

both. 

(  )  9.  Others.  (Please  state  below). 

Others: 

Part  3 .  Additional  Comments:  (Your  comments  on  performance 
risk  or  management  are  solicited.  Please  use  the  back  side 
if  necessary.) 
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Questionnaire  for  Supportabi 1 itv  Risk  Identification 


Part  1.  Based  on  your  experiences  and  understanding,  if 
current  acquisition  variables  (resource  availability, 
pol icies ,etc . )  remain  unchanged,  how  will  it  affect 
supportabi 1 ity  risk  in  major  defense  systems? 

(Definition  of  supportabi 1 ity  risk:  Likelihood  that  defined 
suitability  such  as  supportabi 1 ity,  reliability, 
maintainability,  system  safety  will  not  be  achieved.) 

(Please  check  one) 

Very  High  High  Moderate  Low  Very  Low 


Part  2.  If  your  answer  in  Part  1  was  Moderate,  High,  or 
Very  High,  please  rank  the  candidate(s)  which  is  (are) 
likely  the  risk  driver(s)  from  the  list  below.  You  may 
check  as  many  candidates  as  you  wish  using  the  numbers  1,  2, 
3,  4,  5,  6,  7,  8,  9  ,10,  11  where  1  represents  the  most 
critical  driver  and  11  represents  the  least  critical  diver. 
Do  not  answer  this  question  if  your  answer  in  Part  1  was  Low 
or  Very  Low. 

EXAMPLE:  (  3  )  1 .  ***********^** 

(  2  )  2.********’“***** 

(  )  1.  Contractor  is  given  responsibility  to  develop  the 

Logistics  Support  Analysis  program. 

(  )  2.  Logistics  related  design  parameters  are 

established  after  other  performance  parameters. 

(  )  3.  Manpower  and  skill  analyses  are  based  on 

experience  from  previous  systems. 

(  )  4.  Ambiguous  and  or  unreasonable  requirements. 

(  )  5.  Training  materials  and  equipment  are  developed 

based  on  initial  training  used  for  Test  and  Evaluation 
personnel;  which  are  not  adequate  for  field  use  due  to 
different  skill  levels,  etc. 

(  )  6.  Spares  are  provisioned  during  the  development 

phase  (resulting  in  incompatibility  with  field 
requirements),  or  initial  spares  procured  by  part  number 
without  reference  to  the  specification  (increased 
maintenance  burden). 

(  )  7.  Technical  manuals  are  written  using  the  EMD 


116 


logistics  support  analysis  results  which  do  not  reflect  the 
production  configuration. 

(  )  8.  Inexperienced  and/or  untrained  personnel. 

(  )  9.  Tight  schedule  and  insufficient  manpower  to 

implement  the  relevant  tasks. 

(  )  10.  Poor  program  management. 

(  )  11.  Others.  (Please  state  below). 

Others : 


Part  3.  Additional  Comments:  (Your  comments  on 
supportability  risk  or  management  are  solicited.  Please  use 
the  back  side  if  necessary.) 
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Questionnaire  for  Life-Cycle  Cost  Risk  Identification 


Part  1 .  Based  on  your  experiences  and  understanding,  if 
current  acquisition  variables  (resource  availability, 
policies, etc. )  remain  unchanged,  how  will  it  affect  life- 
cycle  cost  risk  in  major  defense  systems? 

(Definition  of  life-cycle  cost  risk:  Likelihood  of  budget 
overrun  during  system  life-cycle  period.) 

(Please  check  one) 

Very  High  High  Moderate  Low  Very  Low 


Part  2.  If  your  answer  in  Part  1  was  Moderate,  High,  or 
Very  High,  please  rank  the  candidate(s)  which  is  (are) 
likely  the  risk  driver(s)  from  the  list  below.  You  may 
check  as  many  candidates  as  you  wish  using  the  niunbers  1,  2, 
3,  4,  5,  6,  7,  8  where  1  represents  the  most  critical  driver 
and  8  represents  the  least  critical  diver.  Do  not  answer 
this  question  if  your  answer  in  Part  1  was  Low  or  Very  Low. 

EXAMPLE;  (  3  )  i . **^**********^ 

(  2  )  2.*’************* 

(  )  1.  Ambiguous  requirements  or  frequent  requirement 

changes  which  cause  major  engineering  changes  and/or 
redesigns . 

(  )  2.  Inadequate  Logistics  related  programming  which 

results  in  a  high  maintenance  burden. 

(  )  3.  New  technology  is  introduced  without  trade 

studies  being  fully  conducted. 

(  )  4.  Overall  test  program  is  not  implemented 

appropriately  which  results  in  rework  or  an  increase  in 
maintenance  burden. 

(  )  5.  Sole-source  with  no  incentive  to  control  cost 

overrun  for  the  program  or  poor  cost  management /control 
techniques . 

(  )  6.  Tight  schedule  and  insufficient  manpower  to 

implement  the  relevant  tasks. 

(  )  7.  Improper  budget  control  system  which  results  in  a 

poor  understanding  of  the  cost  distribution. 

(  )  8.  Others.  (Please  state  below). 
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others : 


Part  3.  Additional  Conunents:  (Your  comnents  on  life-cycle- 
cost  risk  or  management  are  solicited.  Please  use  the  back 
side  if  necessary.) 
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Questionnaire  for  Schedule  Risk  Identification 


Part  1.  Based  on  your  experiences  and  understanding,  if 
current  acquisition  variables  (resource  availability, 
pol icies ,etc. )  remain  unchanged,  how  will  it  affect  schedule 
risk  in  major  defense  systems? 

(Definition  of  schedule  risk:  Likelihood  of  late  product 
del ivery . ) 

(Please  check  one) 

Very  High  High  Moderate  Low  Very  Low 


Part  2.  If  your  answer  in  Part  1  was  Moderate,  High,  or 
Very  High,  please  rank  the  candidate(e)  which  is  (are) 
likely  the  risk  driver(s)  from  the  list  below.  You  may 
check  as  many  candidates  as  you  wish  using  the  numbers  1,  2, 
3,  4,  5,  6,  7,  8,  9  where  1  represents  the  most  critical 
driver  and  9  represents  the  least  critical  diver.  Do  not 
answer  this  question  if  your  answer  in  Part  1  was  Low  or 
Vei y  Low. 

EXAMPLE:  (  3  )  1 . ************** 

(  2  )  2.************** 

(  )  1.  Ambiguous  requirements  or  requirements  changed 

frequently  which  cause  major  engineering  changes  and/or 
redesigns . 

(  )  2.  Poor  configuration  control. 

(  )  3.  Poor  subcontractor  management. 

(  )  4.  Work  Breakdown  Structure  is  not  being  used  to 

control  the  individual  projects  and  the  integration 
progress . 

(  )  5.  New  technology  is  introduced  without  trade-off 

studies,  or  technical  barriers  impedance. 

(  )  6.  Overall  test  program  is  not  implemented 

appropriately  or  timely  which  results  in  rework. 

(  )  7.  Tight  schedule  and  insufficient  manpower  to 

implement  the  relevant  tasks  properly. 

(  )  8.  Late  involvement  of  production  and  manufacturing 

engineering. 
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(  )  9.  Others.  (Please  state  below). 

Others: 

Part  3 .  Additional  Comments:  (Your  comments  on  schedule 
rislc  or  management  are  solicited.  Please  use  the  back  side 
if  necessary.) 
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Questionnaire  for  Management  Risk  Identification 


Part  1.  Based  on  your  experiences  and  understanding,  if 
current  acquisition  variables  (resource  availability, 
pol icies , etc . )  remain  unchanged,  how  will  it  affect 
management  risk  in  major  defense  systems? 

(Definition  of  management  risk:  Likelihood  of  instability 
and  problems  with  program  management.) 

(Please  check  one) 

Very  High  High  Moderate  Low  Very  Low 


Part  2.  If  your  answer  in  Part  1  was  Moderate,  High,  or 
Very  High,  please  rank  the  candidate(s)  which  is  (are) 
likely  the  risk  driver(s)  from  the  list  below.  You  may 
check  as  many  candidates  as  you  wish  using  the  numbers  1,  2, 
3,  4,  5,  6,  7,  8  where  1  represents  the  most  critical  driver 
and  8  represents  the  least  critical  diver.  Do  not  answer 
this  question  if  your  answer  in  Part  1  was  Low  or  Very  Low. 

EXAMPLE:  (  3  )  1.************** 

(  2  )  2.************** 

(  )  1.  Ambiguous  requirements  or  frequent  requirement 

changes  which  severely  complicate  program  management. 

(  )  2.  Ambiguous  lines  of  responsibility  and  authority 

defined  between  the  ROCAF,  MND  and  CSIST. 

(  )  3.  Risk  management  plan  has  not  been  conceived  and 

integrated  into  the  program  management  strategy. 

(  )  4.  No  advanced  planning  given  for  career 

progressions  or  promotional  opportunities. 

(  )  5.  Industry  and  users  (or  SPOs)  have  no  mutual 

understanding  of  program  objectives  and  its  management, 
which  seriously  complicates  program  management. 

(  )  6.  Lack  of  relevant  management  skills  and  knowledge. 

(  )  7.  Organizational  problems  such  as  bureaucracy,  lack 

of/poor  leadership,  power  struggles,  etc. 

(  )  8.  Others.  (Please  state  below). 

Others : 

Part  3 .  Additional  Comments:  (Your  comments  on  management 
risk  are  solicited.  Please  use  the  back  side  if  necessary.) 
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CASE 

VERY  HIGH 

HIGH 

MODERATE 

LOW 

1 

0.0000 

1.0000 

0.0000 

0.0000 

2 

0.0000 

1.0000 

0.0000 

0.0000 

3 

0.0000 

1.0000 

0.0000 

0.0000 
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0.0000 
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26 
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28 

0.0000 
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0.0000 

1.0000 

37 
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VERY  LOW 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
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0.0000 

1.0000 

0.0000 

47 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

48 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

49 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

50 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

51 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

52 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

53 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

54 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 
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Appendix  Dt  Ranked  Data  for  Performance  Risk  Drivers 


CASE 

ONE 

TWO 

THREE 

FOUR 

FIVE 

SIX 

SEVEN 

1 

4.0000 

2.0000 

3.0000 

0.0000 

0.0000 

0.0000 

1.0000 

2 

0.0000 

1.0000 

0.0000 

2.0000 

3.0000 

0.0000 

4.0000 

3 

2.0000 

3.0000 

5.0000 

6.0000 

1.0000 

0.0000 

4.0000 

4 

0.0000 

3.0000 

4.0000 

2.0000 

9.0000 

8.0000 

1.0000 

5 

0.0000 

0.0000 

0.0000 

2.0000 

3.0000 

0.0000 

0.0000 

6 

0.0000 

0.0000 

0.0000 

4.0000 

3.0000 

0.0000 

2.0000 

7 

1.0000 

5.0000 

6.0000 

7.0000 

2.0000 

4.0000 

9.0000 

8 

5.0000 

6.0000 

4.0000 

7.0000 

3.0000 

0.0000 

1.0000 

9 

5.0000 

8.0000 

9.0000 

3.0000 

1.0000 

7.0000 

2.0000 

10 

3.0000 

8.0000 

1.0000 

4.0000 

6.0000 

2.0000 

7.0000 

11 

2.0000 

1.0000 

2.0000 

3.0000 

4.0000 

1.0000 

6.0000 

12 

7.0000 

8.0000 

5.0000 

6.0000 

1.0000 

3.0000 

2.0000 

13 

3.0000 

4.0000 

5.0000 

6.0000 

9.0000 

1.0000 

2.0000 

14 

0.0000 

0.0000 

3.0000 

0.0000 

2.0000 

1.0000 

0.0000 

15 

1.0000 

2.0000 

3.0000 

3.0000 

2.0000 

1.0000 

3.0000 

16 

1.0000 

3.0000 

1.0000 

2.0000 

1.0000 

1.0000 

2.0000 

17 

2.0000 

0.0000 

4.0000 

0.0000 

0.0000 

0.0000 

1.0000 

18 

3.0000 

6.0000 

7.0000 

5.0000 

8.0000 

1.0000 

4.0000 

19 

1.0000 

2.0000 

0.0000 

2.0000 

0.0000 

2.0000 

0.0000 

20 

3.0000 

0.0000 

0.0000 

0.0000 

0.0000 

4.0000 

2.0000 

21 

1.0000 

2.0000 

6.0000 

3.0000 

4.0000 

5.0000 

6.0000 

22 

1.0000 

1.0000 

1.0000 

2.0000 

2.0000 

2.0000 

2.0000 

23 

2.0000 

1.0000 

3.0000 

4.0000 

5.0000 

8.0000 

6.0000 

24 

7.0000 

8.0000 

5.0000 

6.0000 

1.0000 

3.0000 

2.0000 

25 

0.0000 

0.0000 

0.0000 

0.0000 

2.0000 

0.0000 

1.0000 

26 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

2.0000 

27 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

2.0000 

28 

5.0000 

0.0000 

0.0000 

4.0000 

3.0000 

0.0000 

2.0000 

29 

2.0000 

8.0000 

9.0000 

7.0000 

1.0000 

6.0000 

3.0000 

30 

4.0000 

0.0000 

0.0000 

0.0000 

1.0000 

2.0000 

0.0000 

31 

3.0000 

0.0000 

0.0000 

5.0000 

0.0000 

4.0000 

1.0000 

32 

1.0000 

0.0000 

0.0000 

0.0000 

3.0000 

0.0000 

2.0000 

33 

0.0000 

0.0000 

0.0000 

0.0000 

1.0000 

3.0000 

2.0000 

34 

1.0000 

2.0000 

2.0000 

2.0000 

1.0000 

2.0000 

2.0000 

35 

0.0000 

0.0000 

0.0000 

0.0000 

3.0000 

2.0000 

1.0000 

36 

0.0000 

0.0000 

0.0000 

2.0000 

0.0000 

0.0000 

1.0000 

37 

2.0000 

2.0000 

3.0000 

3.0000 

2.0000 

3.0000 

1.0000 

38 

2.0000 

2.0000 

3.0000 

2.0000 

2.0000 

1.0000 

1.0000 

39 

0.0000 

0.0000 

0.0000 

3.0000 

1.0000 

2.0000 

0.0000 

40 

1.0000 

0.0000 

0.0000 

0.0000 

3.0000 

4.0000 

5.0000 

41 

1.0000 

5.0000 

4.0000 

8.0000 

3.0000 

9.0000 

6.0000 

42 

0.0000 

0.0000 

0.0000 

3.0000 

1.0000 

2.0000 

0.0000 

43 

1.0000 

0.0000 

0.0000 

0.0000 

3.0000 

4.0000 

5.0000 

44 

1.0000 

2.0000 

3.0000 

1.0000 

2.0000 

1.0000 

1.0000 
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Ranked  Data  for  Performance  Risk  Drivers  (Cont) 


45 

6.0000 

7.0000 

5.0000 

9.0000 

4.0000 

3.0000 

2.0000 

46 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

2.0000 

3.0000 

47 

0.0000 

2.0000 

3.0000 

0.0000 

1.0000 

5.0000 

4.0000 
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CASE  EIGHT  NINE  TEN 

1  5.0000  6.0000  0.0000 

2  0.0000  0.0000  0.0000 

3  7.0000  8.0000  0.0000 

4  6.0000  7.0000  0.0000 

5  1.0000  0.0000  0.0000 

6  1.0000  0.0000  0.0000 

7  3.0000  8.0000  1.0000 

8  2.0000  0.0000  0.0000 

9  4.0000  6.0000  0.0000 

10  5.0000  9.0000  0.0000 

11  1.0000  5.0000  0.0000 

12  4.0000  9.0000  0.0000 

13  8.0000  7.0000  0.0000 

14  0.0000  0.0000  0.0000 

15  2.0000  1.0000  0.0000 

16  1.0000  1.0000  0.0000 

17  3.0000  0.0000  5.0000 

18  2.0000  9.0000  0.0000 

19  3.0000  0.0000  0.0000 

20  1.0000  0.0000  0.0000 

21  8.0000  7.0000  0.0000 

22  1.0000  2.0000  3.0000 

23  7.0000  9.0000  0.0000 

24  4.0000  9.0000  0.0000 

25  3.0000  0.0000  0.0000 

26  0.0000  0.0000  0.0000 

27  0.0000  0.0000  1.0000 

28  1.0000  0.0000  0.0000 

29  4.0000  5.0000  0.0000 

30  3.0000  0.0000  0.0000 

31  2.0000  0.0000  0.0000 

32  0.0000  0.0000  0.0000 

33  0.0000  0.0000  0.0000 

34  2.0000  2.0000  0.0000 

35  1.0000  0.0000  0.0000 

36  3.0000  0.0000  0.0000 

37  3.0000  1.0000  0.0000 

38  2.0000  3.0000  0.0000 

39  0.0000  0.0000  0.0000 

40  2.0000  0.0000  0.0000 

41  2.0000  7.0000  0.0000 

42  0.0000  0.0000  0.0000 

43  2.0000  0.0000  0.0000 

44  1.0000  2.0000  0.0000 

45  1.0000  8.0000  0.0000 

46  0.0000  0.0000  0.0000 

47  0.0000  0.0000  0.0000 
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Appendix  E;  Ranked  Data  for  Performance  Risk  Drivers  after 

Ties 


CASE 

ONE 

IWO 

THEiEE 

FOIR 

FIVE 

SIX 

SEVEN 

1 

4.0000 

2.0000 

3.0000 

8.5000 

8.5000 

8.5000 

1.0000 

2 

7.5000 

1.0000 

7.5000 

2.0000 

3.0000 

7.5000 

4.0000 

3 

2.0000 

3.0000 

5.0000 

6.0000 

1.0000 

9.5000 

4.0000 

4 

5.0000 

3.0000 

4.0000 

2.0000 

9.0000 

8.0000 

1.0000 

5 

7.0000 

7.0000 

7.0000 

2.0000 

3.0000 

7.0000 

7.0000 

6 

7.5000 

7.5000 

7.5000 

4.0000 

3.0000 

7.5000 

2.0000 

7 

1.0000 

6.0000 

7.0000 

8.0000 

3.0000 

5.0000 

10.000 

8 

5.0000 

6.0000 

4.0000 

7.0000 

3.0000 

9.0000 

1.0000 

9 

5.0000 

8.0000 

9.0000 

3.0000 

1.0000 

7.0000 

2.0000 

10 

3.0000 

8.0000 

1.0000 

4.0000 

6.0000 

2.0000 

7.0000 

11 

4.5000 

2.0000 

4.5000 

6.0000 

7.0000 

2.0000 

9.0000 

12 

7.0000 

8.0000 

5.0000 

6.0000 

1.0000 

3.0000 

2.0000 

13 

3.0000 

4.0000 

5.0000 

6.0000 

9.0000 

1.0000 

2 . 0000 

14 

7.0000 

7.0000 

3.0000 

7.0000 

2.0000 

1.0000 

7.0000 

15 

2.0000 

5.0000 

8.0000 

8.0000 

5.0000 

2.0000 

8.0000 

16 

3.5000 

9.0000 

3.5000 

7.5000 

3.5000 

3.5000 

7.5000 

17 

2.0000 

8.0000 

4.0000 

8.0000 

8.0000 

8.0000 

1.0000 

18 

3.0000 

6.0000 

7.0000 

5.0000 

8.0000 

1.0000 

4.0000 

19 

1 . 0000 

3.0000 

8.0000 

3.0000 

8.0000 

3.0000 

8.0000 

20 

3.0000 

7.5000 

7.5000 

7.5000 

7.5000 

4.0000 

2.0000 

21 

1.0000 

2.0000 

6.5000 

3.0000 

4.0000 

5.0000 

6.5000 

22 

2.5000 

2.5000 

2.5000 

7.0000 

7.0000 

7.0000 

7.0000 

23 

2.0000 

1.0000 

3.0000 

4.0000 

5.0000 

8.0000 

6.0000 

24 

7.0000 

8.0000 

5.0000 

6.0000 

1.0000 

3.0000 

2.0000 

25 

7.0000 

7.0000 

7.0000 

7.0000 

2.0000 

7.0000 

1.0000 

26 

6.5000 

1.0000 

6.5000 

6.5000 

6.5000 

6.5000 

2.0000 

27 

6.5000 

6.5000 

6.5000 

6.5000 

6.5000 

6.5000 

2.0000 

28 

5.0000 

8.0000 

8.0000 

4.0000 

3.0000 

8.0000 

2.0000 

29 

2.0000 

8.0000 

9.0000 

7.0000 

1.0000 

6.0000 

3.0000 

30 

4.0000 

7.5000 

7.5000 

7.5000 

1.0000 

2.0000 

7.5000 

31 

3.0000 

8.0000 

8.0000 

5.0000 

8.0000 

4.0000 

1.0000 

32 

1.0000 

7.0000 

7.0000 

7.0000 

3.0000 

7.0000 

2.0000 

33 

7.0000 

7.0000 

7.0000 

7.0000 

1.0000 

3.0000 

2.0000 

34 

1 . 5000 

5.0000 

5.0000 

5.0000 

1.5000 

5.0000 

5.0000 

35 

7.5000 

7.5000 

7.5000 

7.5000 

4.0000 

3.0000 

1.5000 

36 

7.0000 

7.0000 

7.0000 

2.0000 

7.0000 

7.0000 

1.0000 

37 

4.0000 

4.0000 

7.5000 

7.5000 

4.0000 

7.5000 

1.5000 

38 

5.0000 

5.0000 

8.5000 

5.0000 

5.0000 

1.5000 

1.5000 

39 

7.0000 

7.0000 

7.0000 

3.0000 

1.0000 

2.0000 

7.0000 

40 

1.0000 

8.0000 

8.0000 

8.0000 

3.0000 

4.0000 

5.0000 

41 

1.0000 

5.0000 

4.0000 

8.0000 

3.0000 

9.0000 

6.0000 

42 

7.0000 

7.0000 

7,0000 

3.0000 

1.0000 

2.0000 

7.0000 

43 

1.0000 

8.0000 

8.0000 

8.0000 

3.0000 

4.0000 

5.0000 

44 

3.0000 

7.0000 

9.0000 

3.0000 

7.0000 

3.0000 

3.0000 
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Ranked  Data  for  Performance  Risk  Drivers  after  Ties  (Cont) 


45  6.0000 

46  1.0000 

47  8.0000 


7.0000 

7.0C00 

2.0000 


5.0000 

7.0000 

3.0000 


9.0000 

7.0000 

8.0000 


4.0000 

7.0000 

1.0000 


3.0000 

2.0000 

5.0000 


2.0000 

3.0000 

4.0000 
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T 


CASE 

EICSHr 

NINE 

TEN 

1 

5.0000 

6.0000 

8.5000 

2 

7.5000 

7.5000 

7.5000 

3 

7.0000 

8.0000 

9.5000 

4 

6.0000 

7.0000 

10.000 

5 

1.0000 

7.0000 

7.0000 

6 

1.0000 

7.5000 

7.5000 

7 

4.0000 

9.0000 

1.0000 

8 

2.0000 

9.0000 

9.0000 

9 

4.0000 

6.0000 

10.000 

10 

5.0000 

9.0000 

10.000 

11 

2.0000 

8.0000 

10.000 

12 

4.0000 

9.0000 

10.000 

13 

8.0000 

7.0000 

10.000 

14 

7.0000 

7.0000 

7.0000 

15 

5.0000 

2.0000 

10.000 

16 

3.5000 

3.5000 

10.000 

17 

3.0000 

8.0000 

5.0000 

18 

2.0000 

9.0000 

10.000 

19 

5.0000 

8.0000 

8.0000 

20 

1.0000 

7.5000 

7.5000 

21 

9.0000 

8.0000 

10.000 

22 

2.5000 

7.0000 

10.000 

23 

7.0000 

9.0000 

10.000 

24 

4.0000 

9.0000 

10.000 

25 

3.0000 

7.0000 

7.0000 

26 

6.5000 

6.5000 

6.5000 

27 

6.5000 

6.5000 

1.0000 

28 

1.0000 

8.0000 

8.0000 

29 

4.0000 

5.0000 

10.000 

30 

3.0000 

7.5000 

7.5000 

31 

2.0000 

8.0000 

8.0000 

32 

7.0000 

7.0000 

7.0000 

33 

7.0000 

7.0000 

7.0000 

34 

5.0000 

5.0000 

10.000 

35 

1.5000 

7.5000 

7.5000 

36 

3.0000 

7.0000 

7.0000 

37 

7.5000 

1.5000 

10.000 

38 

5.0000 

8.5000 

10.000 

39 

7.0000 

7.0000 

7.0000 

40 

2.0000 

8.0000 

8.0000 

41 

2.0000 

7.0000 

10.000 

42 

7.0000 

7.0000 

7.0000 

43 

2.0000 

8.0000 

8.0000 

44 

3.0000 

7.0000 

10.000 

45 

1.0000 

8.0000 

10.000 

46 

7.0000 

7.0000 

7.0000 

47 

8.0000 

8.0000 

8.0000 
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Appendix  F;  Suinmarlaed  Data  for  Supportablllty  Risk 
Maonltudeg  Rated  bv  Respondents 


CASE  VERY  HIGH 

1  0.0000 

2  0.0000 

3  0.0000 

4  0.0000 

5  0.0000 

6  0.0000 

7  1.0000 

8  0.0000 

9  0.0000 

10  1.0000 

11  1.0000 

12  0.0000 

13  0.0000 

14  0.0000 

15  0.0000 

16  0.0000 

17  0.0000 

18  1.0000 

19  0.0000 

20  0.0000 

21  0.0000 

22  0.0000 

23  0.0000 

24  0.0000 

25  0.0000 

26  0.0000 

27  0.0000 

28  0.0000 

29  0.0000 

30  0.0000 

31  0.0000 

32  0.0000 

33  0.0000 

34  0.0000 

35  0.0000 

36  0.0000 

37  0.0000 

38  0.0000 

39  0.0000 

40  0.0000 

41  0.0000 

42  0.0000 

43  0.0000 


HIGH 

MODERATE 

1.0000 

0.0000 

0.0000 

1.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

1.0000 

1.0000 

0.0000 

1.0000 

0.0000 

0.0000 

1.0000 

1.0000 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

1.0000 

1.0000 

0.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

1.0000 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

1.0000 

LOW 

VERY  LOW 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 
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SmBmari2ed  Data  for  SuDDortabl 1 1 tv  RiBk  Maonltudeg  Rated  bv 

ReBDondentB  (Cont) 


44 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

45 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

46 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

47 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

48 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

49 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

50 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

51 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

52 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

53 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

140 


Appendix  Gi  Ranked  Data  for  Supportabllltv  Risk  Drivers 


Case  One 

TWO 

Three  Four 

Five 

1 

5 

1 

2 

2 

1 

2 

3 

7 

6 

5 

2 

4 

10 

8 

9 

6 

7 

5 

6 

2 

7 

4 

6 

5 

10 

7 

8 

2 

5 

9 

10 

10 

1 

2 

8 

9 

11 

12 

5 

1 

6 

2 

13 

10 

6 

5 

7 

8 

14 

4 

2 

3 

15 

2 

2 

2 

1 

2 

16 

1 

2 

1 

1 

2 

17 

5 

3 

4 

18 

4 

3 

5 

7 

8 

19 

1 

2 

20 

21 

1 

2 

5 

6 

5 

22 

2 

1 

2 

3 

3 

23 

5 

1 

6 

2 

24 

1 

3 

2 

4 

25 

6 

5 

26 

1 

2 

2 

27 

2 

3 

28 

3 

4 

1 

29 

4 

30 

10 

9 

1 

8 

7 

31 

3 

32 

7 

5 

6 

1 

2 

33 

3 

4 

1 

34 

3 

4 

35 

2 

36 

3 

37 

3 

38 

3 

39 

3 

1 

2 

2 

2 

40 

1 

2 

3 

1 

2 

41 

3 

2 

42 

3 

1 

43 

10 

6 

5 

1 

7 

Six  Sevoi  Eic^t  Nine  Ten  Elev«i 


4 

4 

1 

1 

4 

8 

1 

3 

3 
2 

4 

1 

1 

9 

2 

1 

7 

1 

9 

1 


6  4  3  7 

3 

3  18 

4  5  2  3 

2 

1  5  3 

9  2  11  3 

3  4 

1  4 

5  4  6  7 

1  1 

7  3  4 

9  3  12 

1  2  5 

12  3  3 

112  2 

1  2 

10  2  6  1 

1  3 

2 

8  4  4  3 

3  12  1 

8  7  3  4 

5 

2  4  3 


1 

2 


1 

5 

3  4 

10  3 

5 

5  6 


1 

3  2 

2  2 

1 

8  9 


4  5 

2 

3  2  1 

6  2  5 

4  2  1 

4  8  9 

2 

1  2 

1 

1  2 

1  2  3 

2 

1  1  1 

2  3  2 

4  5  2 

4  2  3 
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Ranked  Data  for  Supportabllitv  Risk  Drivers  (Conti 


44 

45 

46 


4  5  2 

3  5 
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Appendix  H;  Banked  Data  for  Supportabi 1 1 ty  Risk  Drivers 

after  Ties 


Case  One  Two  Three  Four  Five  Six  Seven  Ei^it  Nine  Tai  Eleven 


1 

5 

9.5 

1 

9.5 

2 

2 

8 

8 

1 

2 

8 

3 

10 

7 

6 

5 

2 

4 

10 

8 

9 

6 

7 

5 

7 

7 

7 

7 

7 

6 

8.5 

8.5 

2 

8.5 

8.5 

7 

4 

6 

5 

10 

7 

8 

8.5 

2 

8.5 

8.5 

5 

9 

8 

8 

8 

8 

8 

10 

10 

1 

2 

8 

9 

11 

7.5 

7.5 

7.5 

7.5 

7.5 

12 

9.5 

5 

1 

6 

2 

13 

10 

6 

5 

7 

8 

14 

5 

8.5 

8.5 

2.5 

4 

15 

6 

6 

6 

2 

6 

16 

3.5 

8.5 

3.5 

3.5 

8.5 

17 

5 

3 

4 

8.5 

8.5 

18 

4 

3 

5 

7 

8 

19 

1.5 

8.5 

8.5 

8.5 

3.5 

20 

7 

7 

7 

7 

7 

21 

1 

2 

6 

8 

7 

22 

6 

2.5 

6 

9 

9 

23 

10.5 

5 

1 

6 

2 

24 

1 

3 

2 

8.5 

4 

25 

8.5 

6 

8.5 

5 

8.5 

26 

1 

2.5 

2.5 

7.5 

7.5 

27 

8.5 

8.5 

8.5 

2 

3 

28 

8 

3 

4 

8 

1 

29 

8.5 

8.5 

8.5 

4 

8.5 

30 

10 

9 

1 

8 

7 

31 

8 

3 

8 

8 

8 

32 

7 

5 

6 

1 

2 

33 

8.5 

3 

4 

8.5 

1 

34 

9 

3 

9 

9 

4 

35 

7 

7 

2 

7 

7 

36 

7.5 

7.5 

7.5 

3 

7.5 

37 

8 

8 

8 

3.5 

8 

38 

7.5 

7.5 

7.5 

7.5 

3 

39 

9.5 

2.5 

6.5 

6.5 

6.5 

40 

1.5 

5.5 

9.5 

1.5 

5.5 

41 

7.5 

7.5 

7.5 

3 

2 

42 

8.5 

3 

1 

8.5 

8.5 

43 

10 

6 

5 

1 

7 

44 

7.5 

7.5 

7.5 

3 

2 

45 

8.5 

3 

1 

8.5 

8.5 

9.5 

6 

4 

3 

7 

9.5 

4 

3 

8 

8 

8 

8 

4 

3 

1 

8 

10 

10 

1 

4 

5 

2 

3 

11 

1 

7 

2 

7 

7 

7 

4 

8.5 

1 

5 

3 

8.5 

8 

9 

2 

11 

3 

1 

1 

3 

8.5 

4 

8.5 

8.5 

3 

8 

8 

1 

4 

2 

3 

5 

4 

6 

7 

11 

3 

7.5 

1.5 

7.5 

1.5 

7.5 

9.5 

9.5 

7 

3 

4 

9.5 

4 

9 

3 

1 

2 

11 

8.5 

8.5 

1 

2.5 

5 

8.5 

2 

2 

6 

10 

10 

10 

3.5 

3.5 

3.5 

8.5 

8.5 

11 

8.5 

8.5 

1 

2 

8.5 

8.5 

9 

10 

2 

6 

1 

11 

3.5 

8.5 

1.5 

8.5 

5 

8.5 

1 

2 

7 

7 

7 

7 

9 

10 

4.5 

4.5 

3 

11 

2.5 

9 

2.5 

6 

2.5 

11 

9 

8 

7 

3 

4 

10.5 

8.5 

8.5 

5 

8.5 

8.5 

8.5 

1 

8.5 

2 

4 

3 

8.5 

7.5 

7.5 

7.5 

7.5 

7.5 

7.5 

1 

8.5 

4 

5 

8.5 

8.5 

8 

8 

8 

2 

8 

8 

5 

8.5 

3 

2 

8.5 

1 

3 

4 

6 

2 

5 

11 

8 

8 

4 

2 

1 

8 

10 

3 

4 

8 

9 

11 

5 

8.5 

2 

8.5 

8.5 

8.5 

5 

6 

1 

2 

9 

9 

7 

7 

1 

7 

7 

7 

7.5 

7.5 

1 

2 

7.5 

7.5 

8 

8 

1 

2 

3.5 

8 

1 

7.5 

7.5 

2 

7.5 

7.5 

9.5 

6.5 

2.5 

2.5 

2.5 

11 

5.5 

5.5 

5.5 

9.5 

5.5 

11 

1 

7.5 

7.5 

7.5 

7.5 

7.5 

8.5 

8.5 

4 

5 

2 

8.5 

8 

9 

4 

2 

3 

11 

1 

7.5 

7.5 

7.5 

7.5 

7.5 

8.5 

8.5 

4 

5 

2 

8.5 
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Ranked  Data  for  SuDDortabi 1 1 tv  Risk  Drivers  after  Ties 

(Cont ) 


46  8.5  8.5  1  4  2  8.5  8.5  3  5  8.5  8.5 
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Appendix  T:  Suiurtzed  Data  for  Lifa-Cycl»  Co«l 

tudes  Rated  by  Respondents 


CASE 

VERY  HIGH 

HIGH 

MODERATE 

LOW 

1 

1.0000 

0.0000 

0.0000 

0.0000 

2 

0.0000 

1.0000 

0.0000 

0.0000 

3 

0.0000 

1.0000 

0.0000 

0.0000 

4 

0.0000 

1.0000 

0.0000 

0.0000 

5 

0.0000 

1.0000 

0.0000 

0.0000 

6 

0.0000 

1.0000 

0.0000 

0.0000 

7 

0.0000 

1.0000 

0.0000 

0.0000 

8 

0.0000 

1.0000 

0.0000 

0.0000 

9 

0.0000 

1.0000 

0.0000 

0.0000 

10 

0.0000 

1.0000 

o.ooor 

0.0000 

11 

1.0000 

0.0000 

0.0000 

0.0000 

12 

0.0000 

0.0000 

1.0000 

0.0000 

13 

1.0000 

0.0000 

0.0000 

0.0000 

14 

0.0000 

1.0000 

0.0000 

0.0000 

15 

0.0000 

1.0000 

0.0000 

0.0000 

16 

0.0000 

1.0000 

0.0000 

0.0000 

17 

0.0000 

1.0000 

0.0000 

0.0000 

18 

1.0000 

0.0000 

0.0000 

0.0000 

19 

0.0000 

1.0000 

0.0000 

0.0000 

20 

0.0000 

1.0000 

0.0000 

0.0000 

21 

0.0000 

1.0000 

0.0000 

0.0000 

22 

1.0000 

0.0000 

0.0000 

0.0000 

23 

0.0000 

0.0000 

1.0000 

0.0000 

24 

0.0000 

1.0000 

0.0000 

0.0000 

25 

0.0000 

0.0000 

1.0000 

0.0000 

26 

0.0000 

0.0000 

1.0000 

0.0000 

27 

0.0000 

0.0000 

0.0000 

1.0000 

28 

0.0000 

0.0000 

0.0000 

1.0000 

29 

0.0000 

1.0000 

0.0000 

0.0000 

30 

0.0000 

0.0000 

1.0000 

0.0000 

31 

0.0000 

1.0000 

0.0000 

0.0000 

32 

0.0000 

0.0000 

1.0000 

0.0000 

33 

0.0000 

1.0000 

0.0000 

0.0000 

34 

0.0000 

0.0000 

1.0000 

0.0000 

35 

0.0000 

0.0000 

1.0000 

0.0000 

36 

0.0000 

1.0000 

0.0000 

0.0000 

37 

0.0000 

1.0000 

0.0000 

0.0000 

38 

0.0000 

0.0000 

0.0000 

1.0000 

39 

0.0000 

1.0000 

0.0000 

0.0000 

40 

0.0000 

0.0000 

1.0000 

0.0000 

41 

0.0000 

0.0000 

1.0000 

0.0000 

42 

0.0000 

0.0000 

0.0000 

1.0000 

43 

0.0000 

0.0000 

1.0000 

0.0000 

gjflt  Jtemi- 


VERY  LOW 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
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Smnmarized  Data  for  Llfe-Cvcle  Coat  Risk  Magnitudes  Rated  by 

Respondents  (Cont) 


44 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

45 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

46 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

47 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

48 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

49 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

50 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

51 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

52 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

53 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

54 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 
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>endlx  J:  Ranked  Data  for  Life-Cycle  Cost  Risk  Drivers 


Case  One 
1  1 


Two  Three 
3  5 


Four  Five  Six  Seven  Eight 
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Ranked  Data  for  Life-Cvcle  Cost  Risk  Drivers  (Cont! 
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Appandlr  Kt  Ranked  Data  for  Life-Cycle  Cost 

after  Ties 

Case  One  Two  Three  Four  Five  Six  Seven 


1 

1 

3 

5 

4 

2 

1 

4 

3 

2 

3 

3 

6.5 

1 

2 

4 

5 

6 

7 

4 

5 

6.5 

6.5 

3 

4 

6 

5 

7 

1 

4 

7 

6 

5 

1 

3 

8 

4 

6.5 

6.5 

3 

9 

7 

7 

2 

4 

10 

1 

2 

4 

6 

11 

3.5 

1.5 

3.5 

5 

12 

5 

6 

1 

7 

13 

1 

6 

2 

5 

14 

6 

2 

1 

6 

15 

3 

3 

3 

6 

16 

2.5 

2.5 

6 

6 

17 

3 

6.5 

6.5 

6.5 

18 

3 

4 

6 

7 

19 

3.5 

6.5 

6.5 

6.5 

20 

6 

2 

1 

6 

21 

3.5 

2 

1 

5.5 

22 

5.5 

5.5 

2 

2 

23 

5 

6 

1 

7 

24 

1 

2 

6.5 

3 

25 

1 

4 

6.5 

6.5 

26 

1 

3 

6 

6 

27 

2.5 

2.5 

5.5 

5.5 

28 

1 

6 

6 

6 

29 

5 

4 

1 

6 

30 

3 

5 

2 

4 

31 

2 

1 

5 

7 

32 

1 

6 

6 

6 

33 

1 

4 

2 

6 

34 

2 

6 

3 

6 

35 

4 

3 

7 

2 

36 

5.5 

5.5 

5.5 

5.5 

37 

1 

3 

6 

6 

38 

1.5 

5.5 

5.5 

1.5 

39 

2 

1 

5.5 

5.5 

40 

1.5 

3.5 

6.5 

6.5 

41 

3 

6 

6 

6 

42 

2 

2 

5.5 

5.5 

43 

4.5 

4.5 

4.5 

4.5 

44 

1 

6 

6 

6 

6  7  2  8 


6.5 

6.5 

6.5 

6.5 

4 

6.5 

6.5 

6.5 

1 

3 

2 

8 

2 

6.5 

1 

6.5 

2 

3 

7 

7 

2 

7 

4 

8 

1 

2 

6.5 

6.5 

1 

5 

3 

7 

7 

5 

3 

8 

7 

7 

1.5 

7 

2 

3 

4 

8 

3 

4 

7 

8 

3 

6 

6 

6 

6 

6 

1 

8 

2.5 

6 

2.5 

8 

1 

2 

4 

6.5 

5 

2 

1 

8 

1.5 

1.5 

3.5 

6.5 

6 

6 

3 

6 

3.5 

5.5 

7 

8 

2 

5.5 

8 

5.5 

2 

3 

4 

8 

4 

6.5 

6.5 

6.5 

2 

6.5 

3 

6.5 

6 

2 

6 

6 

2.5 

2.5 

7 

8 

2 

3 

6 

6 

3 

2 

7 

8 

7.5 

1 

6 

7.5 

4 

3 

6 

8 

3 

2 

6 

6 

5 

3 

7 

8 

1 

6 

6 

6 

5 

1 

7 

7 

2 

5.5 

1 

5.5 

6 

2 

6 

6 

5.5 

3 

5.5 

8 

5.5 

5.5 

5.5 

5.5 

1.5 

6.5 

3.5 

6.5 

2 

6 

1 

6 

2 

5.5 

5.5 

8 

1.5 

7 

1.5 

8 

3 

2 

6 

6 

Bisk  Drivers 


Eight 
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Ranked  Data  for  Life-Cvcle  Cost  Risk  Drivers  after  Ties 

(Cent) 


45  1 

46  1 

47  1 

48  1 

49  1 

50  2 


6 

3 

4 
6 
3 
1 


6  2  6  6 

6  6  6  2 

2  3  5  7 

6  2  6  6 

6  6  6  2 

5.5  5.5  5.5  5.5 


3  6 

6  6 

6  8 

3  6 

6  6 

5.5  5.5 
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Appendi*  T--  Summarigad  Data  for  Schedule  Risk 

Rated  bv  Respondentfl 


CASE 

VERY  HIGH 

HIGH 

MODERATE 

LOW 

1 

1.0000 

0.0000 

0.0000 

0.0000 

2 

0.0000 

1.0000 

0.0000 

0.0000 

3 

0.0000 

1.0000 

0.0000 

0.0000 

4 

0.0000 

1.0000 

0.0000 

0.0000 

5 

0.0000 

0.0000 

1.0000 

0.0000 

6 

0.0000 

1.0000 

0.0000 

0.0000 

7 

0.0000 

1.0000 

0.0000 

0.0000 

8 

0.0000 

1.0000 

0.0000 

0.0000 

9 

0.0000 

1.0000 

0.0000 

0.0000 

10 

0.0000 

1.0000 

0.0000 

0.0000 

11 

1.0000 

0.0000 

0.0000 

0.0000 

12 

0.0000 

1.0000 

0.0000 

0.0000 

13 

1.0000 

0.0000 

0.0000 

0.0000 

14 

0.0000 

1.0000 

0.0000 

0.0000 

15 

0.0000 

1.0000 

0.0000 

0.0000 

16 

0.0000 

0.0000 

1.0000 

0.0000 

17 

0.0000 

1.0000 

0.0000 

0.0000 

18 

1.0000 

0.0000 

0.0000 

0.0000 

19 

0.0000 

1.0000 

0.0000 

0.0000 

20 

0.0000 

1.0000 

0.0000 

0.0000 

21 

0.0000 

1.0000 

0.0000 

0.0000 

22 

0.0000 

1.0000 

0.0000 

0.0000 

23 

0.0000 

1.0000 

0.0000 

0.0000 

24 

0.0000 

0.0000 

1.0000 

0.0000 

25 

0.0000 

1.0000 

0.0000 

0.0000 

26 

0.0000 

1.0000 

0.0000 

0.0000 

27 

0.0000 

0.0000 

1.0000 

0.0000 

28 

0.0000 

0.0000 

0.0000 

1.0000 

29 

0.0000 

1.0000 

0.0000 

0.0000 

30 

0.0000 

0.0000 

1.0000 

0.0000 

31 

0.0000 

1.0000 

0.0000 

0.0000 

32 

0.0000 

0.0000 

1.0000 

0.0000 

33 

0.0000 

1.0000 

0.0000 

0.0000 

34 

0.0000 

0.0000 

1.0000 

0.0000 

35 

0.0000 

0.0000 

1.0000 

0.0000 

36 

0.0000 

0.0000 

0.0000 

1.0000 

37 

0.0000 

1.0000 

0.0000 

0.0000 

38 

0.0000 

0.0000 

0.0000 

1.0000 

39 

0.0000 

1.0000 

0.0000 

0.0000 

40 

0.0000 

1.0000 

0.0000 

0.0000 

41 

0.0000 

0.0000 

0.0000 

1.0000 

42 

0.0000 

0.0000 

0.0000 

1.0000 

43 

0.0000 

0.0000 

1.0000 

0.0000 

MaanitttdeB 


VERY  LOW 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
o.eooo 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
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Sunmarlzed  Data  for  Schedule  Rlgk  Magnitudes  Rated  bv  Re- 

BPondents  (Cont) 


44 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

45 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

46 

o.oroo 

1.0000 

0.0000 

0.0000 

0.0000 

47 

o.ocoo 

0.0000 

1.0000 

0.0000 

0.0000 

48 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

49 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

50 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

51 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

52 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

53 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 
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Case  One  IWo  Three  Four  Five  Six  Seven  Eic^t  Nine 
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Ranked  Data  for  Schedule  Risk  Drivers  (Cont) 


44  2 

45  1  3 

46  3  2 

47  1 

48  1 

49  1 

50  1  4 

51  1 

52  1 

53  1 


1 

12  3 

13  3 

2  3 

7  3  2 

2  3 


3 

2  3  1 

1  3  1 

2 

3  5 

6  8  5 

2  3 

2 


f 
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Appendix  H:  Ranked  Data  for  Schedule  Rlefc  Drivers  after  Ties 

Case  One  TWo  Three  Foxar  Five  Six  Sevoi  ElC^t  Nine 


1 

6 

5 

7 

4 

8.5 

1 

3 

2 

8.5 

2 

1 

7 

7 

2 

3 

4 

7 

7 

7 

3 

3 

4 

5.5 

8 

1 

2 

8 

5.5 

8 

4 

8 

6 

3 

7 

5 

4 

2 

1 

9 

5 

6.5 

6.5 

3 

6.5 

2 

6.5 

1 

6.5 

6.5 

6 

8 

6 

5 

8 

3 

2 

4 

1 

8 

7 

5 

6 

7 

3 

4 

2 

8 

1 

9 

8 

7 

3 

2 

7 

4 

7 

7 

1 

7 

9 

7 

7 

1 

7 

2 

4 

3 

7 

7 

10 

3 

5 

7 

4 

6 

1 

8 

2 

9 

11 

6.5 

2 

6.5 

2 

4.5 

4.5 

8.5 

2 

8.5 

12 

7 

5 

4 

8 

1 

6 

2 

3 

9 

13 

1 

5 

4 

2 

6 

7 

8 

3 

9 

14 

7 

1 

2 

4 

3 

7 

7 

7 

7 

15 

1 

4 

4 

4 

7.5 

7.5 

4 

4 

9 

16 

4 
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6.5 
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1.5 

6.5 
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9 

22 

3 

3 

3 

3 

7 

7 

9 

7 

3 

23 
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3 
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24 

3 

2 

1 
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7 

4 

7 

7 
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25 
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7 
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3 

7 
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Ranked  Data  for  Schedule  Risk  Drivers  after  Ties  (Cont) 
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Appendix  O;  SuaiiMri^^H  Data  for  Management  RIbJc  Maanitudea 

Rated  bv  Respondents 


CASE 

VERY  HIGH 

HIGH 

MODERATE 

LOW 

VERY  LOW 

1 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

2 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

3 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

4 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

5 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

6 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

7 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

8 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

9 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

10 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

11 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

12 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

13 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

14 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

15 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

16 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

17 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

18 

1.0000 

0.0000 

0.0000 

0.0000 

0.0000 

19 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

20 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

21 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

22 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

23 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

24 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

25 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

26 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

27 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

28 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

29 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

30 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

31 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

32 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

33 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

34 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

35 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

36 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

37 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

38 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

39 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

40 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

41 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

42 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

43 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 
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Summarized  Data  for  Management  Risk  Magnitudes  Rated  by 

Respondents  (Cont) 


44 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

45 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

46 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

47 

0.0000 

1.0000 

0.0000 

0.0000 

0.0000 

48 

0.0000 

0.0000 

0.0000 

1.0000 

0.0000 

49 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

50 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

51 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

52 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

53 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 

54 

0.0000 

0.0000 

1.0000 

0.0000 

0.0000 
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Appendix  Pt  Ranked  Data  for  Management  Risk  Drivers 
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Ranked  Data  for  Manaaeiaent  Rlslc  Drivers  (Cont) 
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Appendix  Q:  Ranked  Data  for  Management  Risk  Drivers  after 

Ties 


Case 

One 

Two  Three 

Four 

Five 

Six  Seven 

Eight 
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4 

3 

7.5 
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7 

7 
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4 

8 
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3 

1 

4 

6.5 

15 
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6 

6 
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6 
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8 

16 

3 

3 

3 
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3 

3 

8 

17 

5 

1 

7 

7 

2 

3 

4 

7 

18 
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1 

5 

7 
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2 

3 
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6 

20 
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6 

3 

6 
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7 

8 
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Ranked  Data  for  Management  Risk  Drivers  after  Ties  (Cont) 
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